It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
avatar
Kalanyr: I assume the Warcraft 2 encoding change is probably due to the Blizzard now selling Warcraft II as part of their anniversary stuff. I'll put it on my check into list.
Uh no, I already encountered this issue on the 06/07/2024.
Back then I checked all my serials because Gog had some problem with them.
Only War2 showed this error, tho.
What's the best course of action when you get MD5 or size mismatches? Out of 9245 items I'm getting 33 MD5 mismatches and 28 size mismatches.

I think this started happening this summer with the repo servers were acting weird or they changed the way some of the certs were recorded, but honestly this is over my head.

Is it best to just delete and re-download the mismatched games or is this like a false report?
avatar
guynamedbilly: What's the best course of action when you get MD5 or size mismatches? Out of 9245 items I'm getting 33 MD5 mismatches and 28 size mismatches.

I think this started happening this summer with the repo servers were acting weird or they changed the way some of the certs were recorded, but honestly this is over my head.

Is it best to just delete and re-download the mismatched games or is this like a false report?
I recently had four games in my whole library reporting MD5 mismatches, so I simply tried to run the installers and ran the internal installer check that all multipart GOG games have.

Sure enough, they all reported errors so I deleted the corrupted files and let gogrepoc redownload them. After that they passed the verification.

In my case, I presume they became corrupted because I had some issues with my USB HDD (unplugged it too soon so the !download subfolder became corrupted, so I copied all my GOG installers to several smaller HDDs, reformatted the USB HDD and copied the installers back to it; a mere Windows disc check was not enough, Windows claimed there is nothing wrong with the HDD filesystem even though one of the subfolders was corrupted, Windows even complained about the folder being corrupted if you tried to enter it).
Post edited 5 days ago by timppu
avatar
guynamedbilly: What's the best course of action when you get MD5 or size mismatches? Out of 9245 items I'm getting 33 MD5 mismatches and 28 size mismatches.

I think this started happening this summer with the repo servers were acting weird or they changed the way some of the certs were recorded, but honestly this is over my head.

Is it best to just delete and re-download the mismatched games or is this like a false report?
I haven't been getting any errors, using gogcli instead of gogrepo, not that that should make any difference. So I suspect you do indeed have some corrupt downloads.

Another way to double check, is to use InnoExtract with the test option. which is similar to doing an install but without actually installing. Be aware though, that like install, it is a somewhat lengthy process, depending on the size of the game of course. InnoExtract is a free program, but from what I recall only works on Windows, and maybe in Wine too, but not with SH or PKG or DMG files, which at best can only be checked with something like 7-Zip, outside of using the MD5 checksums.
avatar
Timboli:
innoextract is available on Linux and other unixy operating systems (no wine needed and widely packaged) but as you say it can only test the Windows setup. The .sh files do have a --check option, but it turns out that only checks the installer, not the game being installed :(.
avatar
guynamedbilly: What's the best course of action when you get MD5 or size mismatches? Out of 9245 items I'm getting 33 MD5 mismatches and 28 size mismatches.

I think this started happening this summer with the repo servers were acting weird or they changed the way some of the certs were recorded, but honestly this is over my head.

Is it best to just delete and re-download the mismatched games or is this like a false report?
avatar
timppu: I recently had four games in my whole library reporting MD5 mismatches, so I simply tried to run the installers and ran the internal installer check that all multipart GOG games have.

Sure enough, they all reported errors so I deleted the corrupted files and let gogrepoc redownload them. After that they passed the verification.

In my case, I presume they became corrupted because I had some issues with my USB HDD (unplugged it too soon so the !download subfolder became corrupted, so I copied all my GOG installers to several smaller HDDs, reformatted the USB HDD and copied the installers back to it; a mere Windows disc check was not enough, Windows claimed there is nothing wrong with the HDD filesystem even though one of the subfolders was corrupted, Windows even complained about the folder being corrupted if you tried to enter it).
Interesting and scary. I have them stored on a NAS and it's running Stablebit's scanning and Drivepool tools. I've not gotten any warnings for HDD errors, but I suspect I'll have to do like you and delete each one.
avatar
guynamedbilly: What's the best course of action when you get MD5 or size mismatches? Out of 9245 items I'm getting 33 MD5 mismatches and 28 size mismatches.

I think this started happening this summer with the repo servers were acting weird or they changed the way some of the certs were recorded, but honestly this is over my head.

Is it best to just delete and re-download the mismatched games or is this like a false report?
I wouldn't let this stop you from double checking the integrity of your storage media / moving to new storage media if your existing one is old and has poor recovery capabilities but if you only have a few of these they can just be a consequence of GOG updating the file on the server without changing the name or marking it as updated, in this case the old file will be out of date but isn't necessarily corrupted or non-functional.

GOG has also on rare occasions uploaded files that mismatch there own MD5, whether this has been the file being bad or the MD5 being bad is inconsistent.
avatar
guynamedbilly: Interesting and scary. I have them stored on a NAS and it's running Stablebit's scanning and Drivepool tools. I've not gotten any warnings for HDD errors, but I suspect I'll have to do like you and delete each one.
My belief is that those installer files became corrupted due that USB incident I had, not bitrotting by themselves over time.

Anyway, keeping offline archives of your stuff can be tricky, if you try to make sure none of your stuff becomes corrupted and you can replace the parts which do get corrupted for a reason or another. The first obstacle is how do you even detect something has become corrupted in your vast collection? Gogrepoc at least offers the option of checking against md5 checksums, and then innoextract would theoretically let you test the integrity of all installers even without any checksums, etc.

That's the main reason why I am looking into newer and better filesystems like btrfs because they tend to have stuff like that embedded into the filesystem, so I don't have to keep up some checksum database myself (like gogrepoc does) etc.
Post edited 4 days ago by timppu
avatar
guynamedbilly: Interesting and scary. I have them stored on a NAS and it's running Stablebit's scanning and Drivepool tools. I've not gotten any warnings for HDD errors, but I suspect I'll have to do like you and delete each one.
avatar
timppu: My belief is that those installer files became corrupted due that USB incident I had, not bitrotting by themselves over time.

Anyway, keeping offline archives of your stuff can be tricky, if you try to make sure none of your stuff becomes corrupted and you can replace the parts which do get corrupted for a reason or another. The first obstacle is how do you even detect something has become corrupted in your vast collection? Gogrepoc at least offers the option of checking against md5 checksums, and then innoextract would theoretically let you test the integrity of all installers even without any checksums, etc.

That's the main reason why I am looking into newer and better filesystems like btrfs because they tend to have stuff like that embedded into the filesystem, so I don't have to keep up some checksum database myself (like gogrepoc does) etc.
I was thinking about that too with my server rebuild early this year, but there's no current linux substitutue for handling all the server software I use. Docker seems to be helping a lot, but I'm not sure something like Drivepool would ever work for that.

FYI for anyone curious, some of the errors were caused be my arrogance when I'd changed the folder properties on a few of the games to hidden. I'm convinced the others were caused by something on the backend with GOG. I remember reading about how the cert titles had seemingly spontaneously changed on a few downloads this summer, but I haven't had enough time to dig into all of that. As far as the server stability goes, I've had no reports of any hardware errors or missing data from anyone else or the logs.
avatar
joveian: The .sh files do have a --check option, but it turns out that only checks the installer, not the game being installed :(.
I'm not aware of that. Is that with a Linux program?
avatar
Timboli:
The .sh file GOG distributes is a shell script (text you can look at in an editor, the MS_Check function checks the MD5) followed by a bunch of binary data. The script (if not called with --check that just does the MD5 check) extracts the installer and then executes it. The MD5 is stored in the file itself (as a shell variable) and skips the shell script part of the file. I first assumed it was just checking the entire rest of the file (which seems like it would make sense and not be any harder) but when I looked into having my download script extract and use that value I noticed that it actually just checks a small portion, the same portion that is extracted to get the installer (which is a Linux binary and has both 32 and 64-bit versions). It uses md5sum or the BSD md5 or digest utilities to check the MD5 and dd to extract a gzipped tar file from after the shell script (which is considered to be the first 519 lines). I'm not sure what format the rest of the file is in.
nevermind
Post edited 2 days ago by kenrap
avatar
joveian: The .sh file GOG distributes is a shell script (text you can look at in an editor, the MS_Check function checks the MD5) followed by a bunch of binary data. The script (if not called with --check that just does the MD5 check) extracts the installer and then executes it. The MD5 is stored in the file itself (as a shell variable) and skips the shell script part of the file. I first assumed it was just checking the entire rest of the file (which seems like it would make sense and not be any harder) but when I looked into having my download script extract and use that value I noticed that it actually just checks a small portion, the same portion that is extracted to get the installer (which is a Linux binary and has both 32 and 64-bit versions). It uses md5sum or the BSD md5 or digest utilities to check the MD5 and dd to extract a gzipped tar file from after the shell script (which is considered to be the first 519 lines). I'm not sure what format the rest of the file is in.
Thanks for the info, but I'm not clear on what program you use --check with?
I had a look at InnoExtract again, but that parameter is not listed ... or maybe it is in the Linux version?

I use 7-Zip to test SH files, using the asterisk or hash switch (forget which one). I do that in Windows.
Same story for DMG and PKG files for Mac.
I use 7-Zip, presuming a corrupt container would give an error.
I guess the SH file (etc) is a bit like an EXE file in that it is both script and container.

I wish whatever program(s) that GOG use to build the SH, DMG and PKG files was known, works in Windows, and has a test or check option. NOTE - Of course, because I get the checksum for the installer from the GOG SDK, I don't usually need to test those files with 7-Zip. I only use 7-Zip when the MD5 checksum value is missing.
Post edited Yesterday by Timboli
avatar
Timboli: Thanks for the info, but I'm not clear on what program you use --check with?
I had a look at InnoExtract again, but that parameter is not listed ... or maybe it is in the Linux version?
It's how it's executed with the shell script on Linux as if it were a binary.

avatar
Timboli: I use 7-Zip to test SH files, using the asterisk or hash switch (forget which one). I do that in Windows.
You'd have to open up the shell script with a text editor and remove all the shell script code up to line 519 and change the file extension as "tar.gz" since by then it will become an archive file with the binary data.

avatar
Timboli: I guess the SH file (etc) is a bit like an EXE file in that it is both script and container.
Sort of, yes. On Linux or POSIX operating systems, each file has permission attributes like Windows, but except Windows doesn't have an equivalent of executable permissions bits like on Linux. You could prepend any kind of script (even Python ones) at the very top of the file with a "#!" shebang header to tell the kernel which language interpreter to execute the scripts with. What's not very usual about this shell script is that it has two kinds of binary data to act as two separate files in the same file to do the installation process on Linux. The first set is the tar.gz archive with the installer files, the rest of the binary data is for the installer to handle.

avatar
Timboli: I wish whatever program(s) that GOG use to build the SH, DMG and PKG files was known, works in Windows
If you want to execute shell scripts on Windows, take a look at the MinGW project.
Post edited Yesterday by kenrap