Tell me, what a strange castling names when you restart the script (gogrepo.py update -os windows -lang ru en de)

13:59:55 | ( 73 / 289) fetching game details for elex...
14:00:17 | -> in title "elex" a download has changed name "setup_elex_german_1.0.2946.0_cn_(17315)-1.bin" -> "setup_elex_1.0.2946.0_cn_(17315)-1.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_german_1.0.2946.0_cn_(17315)-1.bin" -> "setup_elex_russian_1.0.2946.0_cn_(17315)-1.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_german_1.0.2946.0_cn_(17315)-2.bin" -> "setup_elex_1.0.2946.0_cn_(17315)-2.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_german_1.0.2946.0_cn_(17315)-2.bin" -> "setup_elex_russian_1.0.2946.0_cn_(17315)-2.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_german_1.0.2946.0_cn_(17315)-3.bin" -> "setup_elex_1.0.2946.0_cn_(17315)-3.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_german_1.0.2946.0_cn_(17315)-3.bin" -> "setup_elex_russian_1.0.2946.0_cn_(17315)-3.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_german_1.0.2946.0_cn_(17315)-4.bin" -> "setup_elex_1.0.2946.0_cn_(17315)-4.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_german_1.0.2946.0_cn_(17315)-4.bin" -> "setup_elex_russian_1.0.2946.0_cn_(17315)-4.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_german_1.0.2946.0_cn_(17315)-5.bin" -> "setup_elex_1.0.2946.0_cn_(17315)-5.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_german_1.0.2946.0_cn_(17315)-5.bin" -> "setup_elex_russian_1.0.2946.0_cn_(17315)-5.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_german_1.0.2946.0_cn_(17315)-6.bin" -> "setup_elex_1.0.2946.0_cn_(17315)-6.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_german_1.0.2946.0_cn_(17315)-6.bin" -> "setup_elex_russian_1.0.2946.0_cn_(17315)-6.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_1.0.2946.0_cn_(17315)-1.bin" -> "setup_elex_german_1.0.2946.0_cn_(17315)-1.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_1.0.2946.0_cn_(17315)-1.bin" -> "setup_elex_russian_1.0.2946.0_cn_(17315)-1.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_1.0.2946.0_cn_(17315)-2.bin" -> "setup_elex_german_1.0.2946.0_cn_(17315)-2.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_1.0.2946.0_cn_(17315)-2.bin" -> "setup_elex_russian_1.0.2946.0_cn_(17315)-2.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_1.0.2946.0_cn_(17315)-3.bin" -> "setup_elex_german_1.0.2946.0_cn_(17315)-3.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_1.0.2946.0_cn_(17315)-3.bin" -> "setup_elex_russian_1.0.2946.0_cn_(17315)-3.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_1.0.2946.0_cn_(17315)-4.bin" -> "setup_elex_german_1.0.2946.0_cn_(17315)-4.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_1.0.2946.0_cn_(17315)-4.bin" -> "setup_elex_russian_1.0.2946.0_cn_(17315)-4.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_1.0.2946.0_cn_(17315)-5.bin" -> "setup_elex_german_1.0.2946.0_cn_(17315)-5.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_1.0.2946.0_cn_(17315)-5.bin" -> "setup_elex_russian_1.0.2946.0_cn_(17315)-5.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_1.0.2946.0_cn_(17315)-6.bin" -> "setup_elex_german_1.0.2946.0_cn_(17315)-6.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_1.0.2946.0_cn_(17315)-6.bin" -> "setup_elex_russian_1.0.2946.0_cn_(17315)-6.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_russian_1.0.2946.0_cn_(17315)-1.bin" -> "setup_elex_german_1.0.2946.0_cn_(17315)-1.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_russian_1.0.2946.0_cn_(17315)-1.bin" -> "setup_elex_1.0.2946.0_cn_(17315)-1.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_russian_1.0.2946.0_cn_(17315)-2.bin" -> "setup_elex_german_1.0.2946.0_cn_(17315)-2.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_russian_1.0.2946.0_cn_(17315)-2.bin" -> "setup_elex_1.0.2946.0_cn_(17315)-2.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_russian_1.0.2946.0_cn_(17315)-3.bin" -> "setup_elex_german_1.0.2946.0_cn_(17315)-3.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_russian_1.0.2946.0_cn_(17315)-3.bin" -> "setup_elex_1.0.2946.0_cn_(17315)-3.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_russian_1.0.2946.0_cn_(17315)-4.bin" -> "setup_elex_german_1.0.2946.0_cn_(17315)-4.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_russian_1.0.2946.0_cn_(17315)-4.bin" -> "setup_elex_1.0.2946.0_cn_(17315)-4.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_russian_1.0.2946.0_cn_(17315)-5.bin" -> "setup_elex_german_1.0.2946.0_cn_(17315)-5.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_russian_1.0.2946.0_cn_(17315)-5.bin" -> "setup_elex_1.0.2946.0_cn_(17315)-5.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_russian_1.0.2946.0_cn_(17315)-6.bin" -> "setup_elex_german_1.0.2946.0_cn_(17315)-6.bin"
14:00:17 | -> in title "elex" a download has changed name "setup_elex_russian_1.0.2946.0_cn_(17315)-6.bin" -> "setup_elex_1.0.2946.0_cn_(17315)-6.bin"

That's interesting. Name changes for those files are determined by MD5 and size matches. In this case the installers for the different languages must be identical. I hadn't considered that possibiliy.
Since you're the first person I've seen who this is relevant for (and if anyone else actually encounters this situation): Do you want to keep different language downloads separately even if they are identical ? Otherwise I can detect this and only record in the manifest the unique versions but log which versions are being skipped (or perhaps generate files name of the form: setup_game_version_lang1_lang2...._langX...(godbuild)-(part).bin . The language order would have to be determinsitic though (probably alphabetical).

Thank you.
Personally, I would more like the possibility of download priorities. Those. if I specify, for example, -lang ru|de|en, then if the game has a Russian version, then only it is downloaded, if there is no Russian, but there is a German one, then only the German version is downloaded, and in the absence of the Russian and German versions the English version is downloaded.
But if it's hard to do, then I can download one language version at a time.

It's certainly doable theoretically. I'll have a think about it while I work on the threading CPU issue.

It doesn't resolve the problem for people who do want all files though, so do you (or anyone else) have any input on that situation ? I'm leaning towards the single download with combined languages name.

This will be optimal in the case when you need to download several language versions at once. Most likely packets in this case differ only in the last files. In Elex 7 files, but only names are exchanged between the first 6.

That's utterly bizarre and sort of defeats combining downloads since they won't work without renaming if that's the case. That leaves me in an awkward position here. I think I'm stuck with checking language as well then.


I guess it makes sense on GOG's side since it minimizes storage space, the identical files probably all point to the same data with the different last chunk stored elsewhere and the installer has the MD5 checksums for all the possible end pieces.

Thank you for that piece of info though.

(I'll still look into a method for specifying language priority of course).

Here's how lgogdownloader handles language (and platform) priority.
It could be useful for you.

Parsing:
https://github.com/Sude-/lgogdownloader/blob/90dcaf0106e4ad1376437d4822d74c7f1b393af1/main.cpp#L470-L471
https://github.com/Sude-/lgogdownloader/blob/90dcaf0106e4ad1376437d4822d74c7f1b393af1/src/util.cpp#L532-L549
Basically it just tokenizes the option string first by priorities ( , ) and then by value combinations ( + ) and stores the priority values in a vector

Filtering:
https://github.com/Sude-/lgogdownloader/blob/90dcaf0106e4ad1376437d4822d74c7f1b393af1/src/gamedetails.cpp#L19-L78
It sets a score for each file using language and platform position in priority vector (lower score is better)
If the file score is higher than best score then remove it from file details vector (download queue)

Ran into a few errors with my latest update.

gogrepo.log

This message appears a few times.
08:41:50 | request failed: <urlopen error [WinError 10054] An existing connection was forcibly closed by the remote host> (3 retries left) -- will retry in 5s...

And once, this message.
08:29:30 | ( 24 / 365) fetching game details for ashes_of_the_singularity...
08:31:21 | request failed: HTTP Error 404: Not Found. will not retry.
08:31:21 | failed to fetch https://www.gog.com/downloads/ashes_of_the_singularity/66543

Those are all normal:
As a rule of thumb:
If something has specified retries and it doesn't use all of them it's functioning as intended (though if you get a lot of connection errors etc you might want to check your Connection / Wi-Fi of course)

Errors that get the same response when you're using the GOG website are on the GOG side (eg I can't download files if GOG doesn't provide them (the usual cause of 404 errors) and I can't get information about items in products where the information isn't given (the usual cause of 403 errors). In the case of the 404, it's nice to know which files are missing but I recommend notifying GOG of the missing file / md5 and asking them to correct it as well. I can code in (optional) suppression of long standing missing things. 403 errors are usually from Kickstarters that haven't provided a promised reward to GOG (this is usually Novellas or Soundtracks) so there's not much anyone can do unless you're a Kickstarter backer and want to chase it up.

(The missing Ashes file is the manual which you can see is missing on the website too)

It would help if update could make use of -skipextras. I'm using gogrepo just to compare installer files to those I already have stored. I don't need it to collect information about the changelog, or the forum url, or the genre, etc...

There is no benefit to skipping that info t's always downloaded incidentally in order to get necessary product information. Extras are what GOG usually calls Goodies so it wouldn't be skipped by -skipextra in any case.

Has anyone noticed gogrepo update slowing down? Last time I ran gogrepo in mid December or so, getting full manifest update for over 1400 games (English Windows versions) took little over one hour, and 15-20 minutes of that was even waiting (to wait for GOG server ban to elapse; the thing about certain amount of http requests per hour etc...).

Anyway, I am running a full update right now, about 1½ hours have gone and "only" about 600 game details have been fetched so far, 800 still to go.

I first tried it with the same older Kalanyr gogrepo version that I used in December (I think a version from early Dec), but I noticed that slower update speed also with it. Today I downloaded the latest Kalanyr version and run it, but this is the update speed I get.

Has something changed on the GOG server side, e.g. they throttle the speed of http requests or something like that? Well, at least it is still faster than with the older vanilla version where it would take like 10 hours to get the file details, but this is still much slower than what I was getting in December. I'd estimate that now it will take me something like 4 hours to get a full manifest update for 1418 games.

Also, checking the log file (while gogrepo is running), something else has certainly changed too: there doesn't appear to be any ban from GOG servers anymore either. So as far as I can tell, it'd appear that GOG has removed that ban where it restricts the amount of http requests per hour, but at the same time they are made slower than before.

I guess this could also be just a coincidence, ie. the GOG servers are just slower now and that is why file details keep coming slower...?

I've noticed the slow down too. I should grab an older version and check if it's a change I made to code or a GOG thing.

I'd say the reason for the lack of ban is that it's highlty unlikely you'd be able to hit the hourly limit at the current speed.

I think it now took something like 3½ hours to do a full update, when it earlier took a little over an hour (part of which was waiting for the ban to be lifted), so yeah much slower now.

I think I noticed this slowdown also with the version I know previously was fast (from early Dec I think), but I didn't run a full update with it this time as I aborted the update, got the newest version of gogrepo and restarted update from scratch. I was hoping that would fix the slowdown, but no.

It is still maybe 3x faster than what it was before the speed up (with e.g. the vanilla version of gogrepo), but considerably slower than in December. Oh well, at least it still works. Is it possible GOG servers are intentionally throttling the rate of serving new http requests, ie. it is not just because the servers are overburdened and can't just serve as fast as before?

Download speeds themselves were ok, I think the games were coming at full speed to me all the time.

EDIT: Yeah and it isn't about my internet connection either, as I switched to my other internet connection as well in between, to make sure it is not about that.

EDIT2: And yeah the lack of ban is probably due to this slow down, as I recall earlier the ban happened around when I reached Wing Commander 4 on the update, and now I wasn't yet anywhere near there during the first hour anyway.

I'm currently looking into what's going on with the mysterious failures to download files in largeish downloads. I think I've found out why it's so fragile, so I should be able to significantly reduce the instance of it happening. But I'll have to wait until I have a big download set to test it again, I used the last big lot with a heap of debugging outputs to work out where things were going wrong.