Yeah I expected it is slower using md5 checksums (if available)... but at least it is foolproof. At least a fallback solution for those Windows installers where md5 is missing.

I don't constantly run the verification anyway, it is enough to run it for the whole library a few times a year, if even that.

I download only Windows installers for now (especially since the "Linux versions" quite often seem to be just the Windows version running on Wine, or the Linux version may be an older version etc.), but I manage them usually on one of my Linux computers. (The installers are on NTFS-formatted USB hard drives which are usable both in Windows and Linux.)

Especially Raspberry Pi4, as it uses very little electricity and is completely silent (=does not need a fan for cooling) so it is perfect for doing tasks that can take even several days, like downloading loads of files, or doing full verifications like this.

To be honest, I wouldn't even consider it as fallback solution. It's not only slow it also doesn't support checking single files. You can only check the entire archive, which in the case of Cyberpunk for example means checking the whole 111 GB. And when something goes wrong and innoextract returns an error code you don't know which file failed. It could be any of the 28.

Then again, if Cyberpunk was missing the md5 checksum (like many games are), what is the option? Then you simply wouldn't know your CP installer is corrupted, until you finally try to install the game with it.

With innoextract at least you can check whether it is ok or not, and if not, then you can inspect more closely which might be the faulty file. Maybe even by trying to install the game (I think at least earlier you would see at which .bin file the installation would fail).

So: it is better than nothing. It is a foolproof method that doesn't depend on e.g. md5 checksums which could just as well be missing or just wrong.

In this thread I've seen some suggest much weirder "solutions" for the missing md5 dilemma, like downloading each and every installer file twice and generating the md5 checksum twice and then comparing the checksums, and if they match, then trust that the file got downloaded correctly and the md5 checksum is correct.

Then again it could just as well be that the installer is already corrupted on GOG servers so then the only way really to check it is to either try to install the game, or check the installer with innoextract.

You don't even need to start installing the game to check which file is bad, just check the option to verify file integrity (only needs to read the files, not write any game data to the file system), and it will report bad files.

Another thing I do routinely is check the signature on the .exe with sigcheck.exe (I happen to do this on a non-windows system using wine). As well as the obvious check to make sure the signature matches the data, I keep a collection of known good signature data (basically the output of sigcheck.exe -q -i) with the variable content filtered out, and use a simple script to find one which matches (gog generates new certificates about once a year or so, which I add to my collection when I start getting no matches on new installers).

EDIT: of course, sigcheck.exe only tests the contents of the .exe, not the .bin files, but every little helps.

My GOG directory is rocking 834 folders at 4.73TB large, if you send me or paste me the inno script for windows i'll run it against mine and cross reference with gogrepoc and see what i can see.

also, I would like to use it as secondary check because I'm paranoid :)

As you may be aware, but some other folks not., my GOGPlus Download Checker program does exactly that.

It also has recursive checking and uses 7-Zip (etc) for other file types like SH (Linux) ... it does however require running under Windows. My program has a floating dropbox for quick drag & drop usage. BIN files can be checked independently, if separated (different folder) from any accompanying EXE file.

As at least one other member has mentioned though, it is a lot slower than a MD5 check using GOG SDK values. For those who don't know, this is because it checks every file inside an EXE or BIN file package (container), rather than just each file package itself.

My program can also be used for downloads from other game places like Itch.io and IndieGala and Humble etc ... and it is not restricted to just game related files.

InnoExtract is certainly worth using for situations that call for it.

The option I'm going with is simply to retry getting the XML files until I have all of them.

The problem is related to the CDN gog has recently switched to. Depending on time and area some XMLs are available while others aren't. So simply retry at different times and from different locations by hopping around the world using a VPN until you catch 'em all.

Welcome to GOG: Pokemon "Gotta scrape them all GOGamon!"

Well, to each his own I guess. That sounds a bit erratic to my taste also because the files keep changing all the time (and then the new XMLs may be missing again etc.). Good thing there there are options I guess.

I think there are some games like "A Golden Wake" (IIRC) which have had the missing XML for like... forever? So the innoextract test can handle those too.

https://github.com/Magnitus-/gogcli/blob/main/sdk/download-async.go#L26
https://github.com/Magnitus-/gogcli/blob/main/sdk/download-async.go#L29
https://github.com/Magnitus-/gogcli/blob/main/sdk/download.go#L203
https://github.com/Magnitus-/gogcli/blob/main/sdk/download.go#L87
https://github.com/Magnitus-/gogcli/blob/main/sdk/download.go#L322

Code could definitely be cleaned up a little at this point in the sdk (I had to do a lot of unexpected crazy workarounds in this part of the code), but anyways, that's it.

You just compute md5 checksum twice only for the files that have bad metadata (once when you generate the manifest, once when you actually download the file and hopefully check that the downloaded checksum matches what is in the manifest). You detect that the file has bad metadata when it returns a 200 for the metadata request, but the body in the response is messed up.

Simplest solution. All installers are properly downloaded and vetted. Works for all installers no matter the type. No crazy hops across the CDN. It is slower, but as long as only a minority of installers suffer from the problem, you just byte the mostly manageable extra delay.

On an update of ~56 games, I had this much so still a minority of installers:

[sdk] Bad metadata for /downloads/severed_steel/en1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/sacred_2_gold/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/solasta_crown_of_the_magister/en1installer5: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/phoenix_point_year_one_edition/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/solasta_crown_of_the_magister/en1installer4: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/neon_abyss_alter_ego_pack/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/firewatch/en1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/dv_rings_of_saturn_anthropogenesis/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/no_mans_sky/en1patch2: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/dead_cells/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/they_always_run/en1patch2: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/hellish_quart/en1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/sacred_2_gold/fr1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/firewatch/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/sacred_2_gold/es1installer1: File metadata was still fetched using much longer workaround method.

If it gets really bad in terms of extra delays, I'd look into other complementary solutions in addition to the above method (CDN hoping in the cloud & possibly innodb verification), but I don't think we're quite there yet (plus those other methods will not work on all installers, so I view them more complementary solutions to speed things up if too many installers need to be downloaded twice as opposed to comprehensive solutions that can stand on their own and work 100% of the time).

In the end, the above method is just that slow, but dependable fallback solution that will always have your back no matter what. So I think regardless of what complementary solutions are implemented to speed things up, the above is pretty much a must if you want to cover all installers dependably.

Mm.. but is this better than simply retrying gogrepo after a while for the titles with temporary errors?

A lot of the other mitigation strategies aren't bad and are even superior in the cases where the mitigation strategy works (in terms of incurred delays for the end user and usage if they don't have unlimited internet).

The problem is that they don't work 100% of the time (and tend to have higher implementation time which is significant for projects with more limited manpower) and when push comes to shove, base functionality is the most important consideration so I think it makes sense to start with that and expand from there.

Don't get me wrong, if someone is interested in exploring innodb or leveraging the cloud to hit different spots in the cdn (I know have a fair amount of interest in the later), then go for it, it is your time. Just realize that if your primary goal is to best leverage your limited time to just to get verified installers working properly, this is probably not the best bang for your buck as I've come to realize.

When I do a gogrepoc update with -skipknown I've noticed in recent months that this not only downloads "new" games in my library, but it also downloads "updated" games. Is this how it has always behaved or is my memory failing me? I thought that -skipknown only downloaded new games and that -updateonly downloaded updated games.

I think e.g. this does it (run in the folder under which all your GOG installer subdirectories are, and the newest innoextract.exe could be there too):

for /F "usebackq" %i in (`dir /s /b *.exe`) do innoextract.exe -t %i

There is quite a lot of output, and for the failing installers the error is: I tried to pipe the output to text files:

for /F "usebackq" %i in (`dir /s /b *.exe`) do innoextract.exe -t %i 1>>ok.txt 2>>error.txt

But then error.txt doesn't list which installers have the problem, only that there are problems... Maybe someone can figure it out better to get better logging.

For Linux, a similar command would be (to verify Windows installers in Linux):

find . -name '*.exe' -exec innoextract -t {} \;