Ok, here are all of the technical solutions I'd suggest using in concert, in their most recent (most considered) forms:
First, if you want to prevent ignorant users from attempting to open the installers the wrong way but you want to appear benevolent, my advice is to apply three protections in concert:
1. Choose a custom file extension like .gogpiece (BIN is heavily overused and, as I remember, VLC associates with it to support certain MPEG-1 era MPEG files)
2. Customize (but don't change the length on) the 7-byte constant header prefix in your RAR files so WinRAR won't recognize them. This should make your unrar.dll simpler to customize and maintain since, unless unrar.dll is built from the most insane code on the planet, it's likely to be simply a matter of editing a single
const char[] line. (Might I suggest replacing the "Rar!" in that header with "Gog!"?)
Here are some lines you can incorporate into your workflow, with or without modification, to enable and disable the protection, respectively:
python -c 'with open("setup_some_game-1.gogpiece", "r+b") as fobj: fobj.write(b"Gog!\x1a\x07\x00")'
python -c 'with open("setup_some_game-1.gogpiece", "r+b") as fobj: fobj.write(b"Rar!\x1a\x07\x00")'
They're simple, they execute in a split second, and they're even harder to mistake for DRM than using the old, proprietary InnoSetup packing which required a custom extractor to be written.
3. Use
appropriate HTTP headers to opt out of Internet Explorer content sniffing as an extra layer of protection, just to be safe:
Content-Type: application/x-gog-binary
X-Content-Type-Options: nosniff
Content-Disposition: attachment; filename=whatever.gogpiece
4. (optional) Write a simple dxdiag-like information-gathering tool to include in GOG installs. Modify the GOG support form so that, to be eligible for post-install support, Windows users must attach the tool's output. Write the tool so it checks for registry keys that only exist if the InnoSetup installer was run. If it can't find them, tell users that their install is broken and post-install support won't be provided until they do it properly.
(In other words, force Windows users requesting support to run an automated version of the "is it plugged in?" step on the troubleshooting checklist. That way, you're not restricting what they paid for... you're just making a rule about the form a support request must take. It's also a good place to put Windows version checks and "is it Wine?" checks if you think users are lying to support to get undeserved help.)
UPDATE: You could also incorporate hash checks for all installed files, if you wanted, so the tool would be able to diagnose missing or corrupt files in the install without users even having to contact GOG.
Second, if you want to extend the digital signature's validity check to the RAR, you need a mechanism that will verify the RAR's validity using something stored within the EXE. (Based on either a secure one-way hashing function or asymmetric crypto so that the tools required to forge it are not available to recipients)
The simplest solution would be to store an SHA1 or SHA256 hash of the RAR in the EXE, but that would prevent the rapid iteration you desire unless you iterate on a debug build without it and then make a release build with it.
The more correct solution (though one which would take longer to implement) would be to include a public key in the EXE and a signed manifest of expected contents in the RAR. That would allow you to change the RAR all you want as long as you have the private key to sign new manifests.
Then, Windows would verify the EXE's signature, the EXE would verify the manifest's signature, and then it could use the manifest to verify that the rest of the RAR's un-encrypted contents hadn't been tampered with.
(Security is hard, because you need to think of everything but they only need to find one thing you missed. However, That "chain of trust" design is used all over the world to protect things. It's how TLS/SSL certs are verified, it's the same approach that things like smartphones and consoles use to make their DRM robust, and the same approach Secure Boot relies on for a verified system. As long as you verify every file and the only C/C++/Pascal you change is the string constant which defines what a RAR header prefix looks like, InnoSetup and UnRAR's mature codebases should be and remain bug-free enough to ensure that attackers can only circumvent the checks by convincing users to disable or ignore them.)
