Just as an update I've been at a 3 day conference thing and am 97% dead so this will likely not be done for a few days. Apologies.

To me your findings seem similar to mine, ie. the GOG servers allow only a certain amount of requests per hour. To you the errors start coming earlier than for me, but it could be simply that you generate more requests per game and faster, depending on the game type or if you are getting more language and/or OS versions of games? Just my assumption...

So far it appears to me the point would be mainly to limit how many requests are made per hour. When a certain limit is reached, apparently the script just needs to wait until one hour has passed since the start of the script, before continuing?

People with less games will apparently be unaffected by this, people start seeing this when they have well over one thousand games (>1300 or so), but again I don't know if the limit is reached earlier with trying to get several language and OS versions at the same time. I might just as well try running update with all language and OS versions to see if the limit is reached earlier, but right now I am running verify on my GOG collection...

Did you change the HTTP_RETRY_DELAY and HTTP_RETRY_COUNT values? As said, I have set both to 30 (30 seconds delay before a retry, and retrying up to 30 times if needed), and I think I got all the game data. I think the default values are quite low for this type of thing, they were probably originally meant for something else.

HTTP_RETRY_DELAY = 30 # in seconds
HTTP_RETRY_COUNT = 30

With these settings, for a couple of games it took up to 12 retries before GOG gave the data (30 seconds delay in between requests), while for some one retry was enough. I have no idea how that works, it is just about luck or does it depend on how many items (files) there are for a game (a game with only few files + extras will need less retries more likely?)

Either way, after one hour has passed, then this become irrelevant. I don't know if it makes sense to try to get the data during the ban period, or just wait it out. In my case I could get data for seven games during the ban period which doesn't really save much of time compared to just waiting.

I wish some cyber security expert or backend developer would comment on these as these are all just my assumptions on how things seem to be working on the server side (e.g. how are the number of http requests normally limited on the server side, and why does it still give data after several retries, is that intentional?).

It's impossible to tell externally because the error code is wrong which means the behaviour is inconsistent. I'll check if the standard request limits header exists, and respect it if it does. Otherwise I'll set a relatively conservative limit of 1100 games per hour before stopping and waiting the reminder of the hour. That should cut down the wait to significantly less than an hour (since it should take a significant portion of an hour to make that many requests in the first place unless your ping is in the low single digits in milliseconds)

Will you make that easily user-configurable (in case different limits work better for different users, and/or GOG changes it at some point)?

Well, I guess an user can always change it somehow, considering it is a python script...

Ish. I'll definitely break it out into a constant at the top. Maybe an override flag or setting file.

Thanks, I will try that, I was using the default settings. To be more precise, 26 games have failed in the last run and have been moved to my orphaned folder after cleanup. The command I use is python gogrepo.py update -os windows -lang en, only Windows and English versions.

Oh, that's bad. I wasn't sure if such games (for which gogrepo fails to get data after several retries) will be merely missing the md5 checksum, but apparently it makes them completely obsolete in gogrepo's eyes (which of course makes sense, it can't get any data for them if there is an issue).

Anyway, fortunately it seems GOG somehow does let one get the data after several retries, as long as one keeps trying long enough. And of course when the one hour limit is reached, then no errors (unless one would reach the limit again, but I presume no one has that many GOG games yet, in fact the GOG store itself doesn't have that many games...).

Frankly, since a game will become orphaned if no data is received, I'd put the HTTP_RETRY_COUNT = 9999, or something like that. :) And the retry delay to something suitable (30 seconds is probably fine, could be even more like 60 seconds) so that you are not constantly hammering the GOG servers while your ban is on (no idea if that matters to anything, but still...).

In a way that would also fix this thing right now, the script would just keep retrying for the first failed game until one hour has passed (since starting the script). So maybe the change to wait until one hour has passed is not necessarily needed after all, dunno...

I just tested a full update (without old manifest file), getting all os versions (windows linux mac) and all languages that I know of (en cz da de es fr it hu nl no pl pt br ro fi sv tr ru ko cn jp).

As I suspected, then the limit is reached much earlier, already at game 786 or thereabouts. So for those who want to get several language and os versions, they have to check themselves where it seems to start hitting them.

However, if you can't find that "request limit header" and you'd consider adding "wait until the full hour is up"... I've now been thinking does it necessarily make sense to implement any extra fix for this? To me it seems there is already a fix or at least a workaround which does almost the same thing: setting the delay and count to something like:

HTTP_RETRY_DELAY = 30 # in seconds
HTTP_RETRY_COUNT = 9999

does pretty much the same thing (I am using those values for this test). When the errors start coming, it will just keep trying until the GOG server lets it continue. Naturally this also works for other cases I guess like if one has a flaky internet connection that loses its connection once in a while and needs to be reinitialized etc. (that has happened to me in the past when I had to reboot the cable modem, so gogrepo update was running for several minutes without an internet connection).

Also, it seems GOG servers still let one get data for games during the ban period, albeit quite slowly and after many retries, so the whole ban period doesn't go fully to waiting.

What do you think? Do you see any downsides in using such high values for delay and count?

I'll see if I can avoid stomping existing manifest entries for these errors too.

Yes. It's incredibly cruel to GOG servers to do that on a large scale. It's fine for individuals but a very bad idea for default setting of a publicly available program. GOG really should not be letting repeat requests through under timeout because it encourages this. The behaviour here is completely bizarre.

Yeah, I wasn't thinking it is a default setting, but rather an instruction to those with big GOG libraries like "if you start seeing the 509 error messages because you have so many games, you can change these values blaa blaa blaa".

And since the retries would come every 30 seconds (or even less often, it could just as well be 60 seconds or more), is it really stressful to the servers, even more stressful than normal gogrepo usage? I agree the GOG servers shouldn't be hammered constantly and repeatedly (like every few seconds or what the default value was) when the ban is on.

Anyway, yeah, if you feel the automatic waiting mechanism is still a preferable idea, by all means go ahead (as long as the trigger value can be easily configured, as in my tests the errors started coming at around 786 or 1352 games, depending how many OS and language versions I was trying to get). And just to note that the problem with big libraries can still be overcome already now by changing those values (without running into Jabba's problem where the manifest file will miss data for many games, and gogrepo clean will consider them as obsolete games because of the missing data).

I did check and GOG doesn't provide rate limit data unfortunately. I'll make a script that counts the actual connection requests per hour and send it to you when I'm feeling less dead, and if you can let me know how many requests it takes specifically (it's one request initially + 1 per page of the library + 1 per game + 0-2 per item within a game but that depends on the item).

Ok I am sure me and/or Jabba can check that, when available (Jabba seems to have the most GOG games :)).

Update on X3 Terran War Pack - With the new 3.4 installer all old patches have been removed. So the missing MD5 on that one file is no longer an issue.

Hmmm. I just did a full manifest update / download (of missing stuff) / verify. And fortunately(?) I had 200 GB to download.

A few things to note:

The Surface Book is genuinely awful at this, not sure if it's not supplying enough power to the USB3.0 port or if it's overly aggressive on saving power for the Wifi (I suspect this one since manifest update is actually reasonably IO light only writing to disk every 50 games) but it takes 6 hours to do a full manifest build. This makes it kind of useless for testing the delays.

The problem with hanging due to GOG stopping sending data is fixed. The issue with threads deciding to randomly go AWOL is distinctly not (this is a problem that existed in the original build too). There's no error message of any kind , the threads just disappear and are marked as dead so the section terminates properly, it's like they try and get stuff from the work pool but it's empty even while they are in the middle of downloading (so I guess there's a fall through error path I'm missing somewhere ?). It looks like it's stopping in mid download then requesting more work then flubbing that too, since 2 or 3 AWOL threads (unsure since one thread seems to have come back right at the end by my log) have stopped 11 items downloading*. This is obviously going to take significantly more testing that is completely at GOG's mercy. If you've got a largish collection that you update infrequently enough to get ~100 GB in downloads and are willing to help testing let me know. I'll have to put together a debugging build for this too.

Very weird note - GOG appears to be randomly having (rare) SSL issues ? Maybe increased server load from the Winter Sale ? I'll have to catch and retry on this exception somehow.

*In the truly bizarre every one of those items had it's pre-allocation done though. So wherever the problem is happening it must be after that.