-Rars are used for convenience, as they have some features that the old archives lack. For example when making a test build of the game, it's faster for us to update the archives than to repack them from scratch when making small changes for testers.
Compared to letting InnoSetup pack them, I can understand that... but why RAR rather than some other standalone compression format with a similar compression ratio and fewer licensing restrictions on the compressor (like 7-zip)?
There were situations, when users would download just a single part of the installer, or try to unrar it manually (because apparently some browsers detect our new archives as rar files), or even try to open the .bin files with the VLC Video Player.
In such a situation I think it's better to give immediate "it won't work that way" message, rather than allow someone to make a "partial" installation, which may or may not work, without any information.
That makes sense... but have you considered just prepending some kind of stub to foil the header detection? That'd give you a lot more control. Heck, you could make it an EXE that just pops up a "Please use this installer properly" dialog.
(That's how self-extractors work, so tools like unrar and 7zip will try to seek past a prepended stub when you ask it to extract an EXE... or image file for that matter.)
Another reason - I want to avoid the situation where someone tampers with the archives (let's say adding malware, or some illegal content), and uploads the modified version on torrents. I don't want the GOG Installer installing anything else than it was supposed to, and it doesn't matter how it was obtained.
Malware pushers tend to be better at this kind of stuff than we are and have more of an incentive and yet we already figured out the password algorithm and realized that innounp.exe will unpack an InnoSetup installer completely enough to modify the resources and repack it into a new installer.
(Speaking of which, are you using an unmodified InnoSetup instance? If so, they could just unpack the InnoSetup EXE, rebuild it with a malware-installing component, and reuse your password-calculating unrar.dll without figuring out how it works.)
We don't really support installing the game by manually unpacking the archives (for whatever reason you do that).
Mostly to run the games on Linux or MacOS... either because it's a DOSBox game, because these new installers crash under Wine (they do), or because the installer contains resources we want to use with a clean-room engine clone like CorsixTH for Theme Hospital.
...plus, it just generally rubs us the wrong way when we don't have the option of unpacking our installers ourselves should GOG ever go under and Wine ever drop backwards compatibility, leaving us with only open-source clean-room engine clones and archive unpackers.
On the other hand, I see you already figured out the algorithm for obtaining the password, so you are still able to do as much. I'm not going to say "Hey, good job hacking into our software guys!", but I'm not going to try and make the password harder either.
Obviously, but I still think you should re-evaluate your approach. It's ineffective at preventing malware distribution, it gives a bad impression to people who don't know your intentions, there are better ways to prevent a browser from mis-identifying an archive, and I can probably think of some alternative proposals for minimizing the chance that Windows users will unpack them rather than running them. Update:
One option would be to try snipping off the magic constant at the beginning of the archive header that identifies it as that file type. ('\x52\x61\x72\x21\x1a\x07\x00' in the case of RAR) Then, even if users do rename them to whatever archive extension they actually are, WinRAR or 7zip or whatever wouldn't recognize them while your custom unrar.dll and our niche extraction tools could easily be modified to act as if offset 0 in the file is actually offset 7 and the check already passed.
After all, your old InnoSetup BIN files already get mis-identified as AIX core dumps half the time.