Is it possible to re-phrase the question?

In addition to the situation where the name and md5/size are the same there's also a situation where the name is the same and the md5 are different in that case it'll automatically rename it. The message is:

' -> ' + update_item.name + ' already exists in this game entry with a different size and/or md5, this file renamed to ' + deDuplicatedName

Do you ever get a message of that form when updating with all languages enabled ?

I'm not sure if I'm doing something wrong OR this is a bug. I decided I would download all Linux installers from my GOG library but skip extras, as I have them from the windows side of my backups. Here's what I'm using:

python gogrepo.py update -lang en -os linux

python gogrepo.py download -skipextras

The above combination downloads nothing at all. So either I'm doing something wrong (probably) or it's broken somehow.
-skipextras should download everything except any goodies right?

EDIT: Should mention I'm using Kalanyrs version

EDIT 2: Never mind, I've figured it out. I have to use gogrepo.py download -os linux -skipextras
not sure why this is required, should it not just download whatever is in the manifest sans extras if the manifest only contains linux builds?

Hello,
The latest Kalanyr script is doing a great job with the manifest download speed. However, I get an 509 error with 500 games left (1754 total). I will use the older script when I'm no longer blocked. I hope you can improve this further. Keep up the good work.
Jabba

The script dont create for games like "Quake III: Gold" who has 2 serials a !serial.txt I hope someone can fix this.

I need more data on what's causing this. So late in the downloads it's probably not the speed the server is hit generally, it's either the total number of requests or some specific games with a looooot of item entries.

I don't get this happening and I've got ~1275 games , so it's hard for me to diagnose this. If people could let me know what number of game is and the game you get blocked on (and the one before).

Also your ping to gog.com

I'll look into this. Is it just Quake ?

Last time I ran a full manifest update twice (because I wasn't sure if something went wrong with the first one). I have total 1408 games, and the first time the 509 errors started with Wing Commander IV, which is the game number 1353 / 1408.

The second time they started a bit later, at Wings of Prey, which is 1360 / 1408.

However, the manifest update did complete eventually, the last few dozen games just took a longer time. I have changed the script values for me so that the HTTP_RETRY_DELAY is 30 seconds, and HTTP_RETRY_COUNT is 20. So it retries many many times if needed, but also it waits longer before retrying, if it receives the 509 error. For some of the games it had to retry a few more times before moving on to the next game.

EDIT: So now that I look at the log more carefully, I started getting 509 errors at game 1360 (Wings of Prey) I think, and received errors also for the next few games. 1367 (Wolfenstein 3D) is the first game after that not receiving the 509 error, and the rest of the games didn't receive it.

If it is a ban and if it is somehow based on time, these are the timestamps:

22:30:26 I start running gogrepo update
23:07:48 The first 509 error (with Wings of Prey Complete)
23:30:39 No more 509 errors, rest of the game data came down fine with update.

EDIT: And to clarify, these happen at least for me with gogrepo update. I don't think I had any issues with gogrepo download (I didn't go through the whole log file though).

EDIT2: I think I will run gogrepo update again tomorrow, just to test this (and there is the King of Dragon Pass remake waiting for me anyway...).

Using gogrepo.py :
md5 : 5d276415c94d3482847eb96fa5d6427b
sha256 : 8dd66ca027a5e10eb6c160e59f17eda332f475bf3f9391795a42bac1ebf77388

... and using the command:
python3 gogrepo.py update -os windows linux mac -lang en cn it cz ar gk es ru nl pt no tr ro pl fr bl de jp hu br fi da ko sv sk sb

I fail to see mention of " file renamed to " in the gogrepo.log
I have less than 200 games, so that is a small sample size of the entire GOG catalog.

There are people who keep Linux and Windows in one manifest but split them on downloads.

On the topic of 509 bans, how long do a they last ? I'm thinking the best way to do this until I can get a lock on the source is to detect a 509, wait the ban period then redo the item that got bannerd and carry on. That lets you continue to run it with confidence unattended and let's us continue to gather data.

I'd also like to note that 509 is completely the wrong error for this situation, the correct error is 429 which also specifies how long to wait.

Well. I have the timestamps in my earlier message, but then I wouldn't have any idea what the limits, if any, are. and whether they are based on time, amount or frequency of requests etc.

If it is based on time, then in my case the errors started pouring in after about 37 minutes of starting the update, and stopped about 23 minutes after that. And after that it was error free till the end (but then there were only a few dozen titles to go through).

I don't know if the rate of requests (while you are under the ban) cares at all how often you keep retrying, e.g. if it continues the ban even further if you keep trying to hammer it? Either way, I felt that limiting the http retry delay to 30 or even 60 seconds is fine, as the server will not let me continue anyway no matter how fast I keep knocking on their door.

Also, I felt the default value for retry count (3?) is far too low, so I topped it up to 20. At least in the first run I recall it varied a lot how many retries it took to succeed. Sometimes three retries was enough, in some case it would reach 10 retries.

Come to think of it, now I am even more confused how exactly does it work. To me it did seem like I was able to get the data for the games when the ban(?) was on, but it would take several retries per game. And after that, the ban would go away and no retries were needed. Does it mean the ban does not block you completely?

Yeah, at first I didn't even know whether it was a ban targeted to me, or if GOG servers were generally under heavy load, and everyone would get those same errors.

I tested it again with the latest gogrepo of yours, with these settings:

# HTTP request settings
HTTP_FETCH_DELAY = 1 # in seconds
HTTP_RETRY_DELAY = 30 # in seconds
HTTP_RETRY_COUNT = 30
HTTP_TIMEOUT = 30

So I increased both the delay and count a lot, as I don't want to hammer GOG servers constantly while being banned, and neither want to give up too easily.

The 509 errors started coming at almost exactly the same place as before, around game 1352 or so (when searching only for English Windows versions of games). These are the exact timestamps:

gogrepo.py update started:

13:44:22 | loading local manifest...

The 509 error comes the first time (after about 40 minutes from the start):

14:24:23 | (1352 / 1408) fetching game details for wing_commander_3_heart_of_the_tiger...
14:24:26 | request failed: 509 Server Error

The first game comes without errors:

14:44:49 | (1359 / 1408) fetching game details for wing_commander_secret_ops...

As said, the games during the ban do seem to get the data too, but only after several retries. For some one retry seemed to be enough (waiting for 30 seconds), while some needed 12 retries (a 30 second wait between each).

Since I don't know if this depends e.g. on max requests per hour, or is it based on time (constant requests for 40 minutes = ban?), would it make sense to add two new parameters to the script, something like:

HTTP_COOLDOWN_AFTER = 1300 # number of games, 0 = never
HTTP_COOLDOWN_PAUSE = 300 # in seconds

or something like that, user-configurable parameters for after how many games there is a cooldown period when no more http requests are sent, and how long that pause should be?

I don't know if this is wildly different for different people based on e.g. what versions they are trying to get, like if someone wants also Linux and Mac versions and several language versions, will they hit the limit much sooner (e.g. after 700 games)? Everyone would have to come up with values that suit their needs, and I guess the cooldown period time just need to be tested, ie. after how long wait you've evaded the incoming ban?

I guess people with only a few hundred games won't even see this issue then.

Or so you have some other idea how this should be handled?


EDIT: Looking at the timestamps, it seems apparent that there is a limit of http requests per hour (in both tests 509 errors were not received anymore exactly after one hour had passed since starting the gogrepo update). So the "cooldown pause" period should be set so that it won't continue until one hour has passed since the start of the update run.

If that was the case, then maybe there should be only one parameter ("http cooldown after"), after which gogrepo stops until one hour has passed since the start of the update run? That way the script would limit the amount of http requests per hour.

What I still don't understand is that GOG doesn't seem to block you completely during the ban, as several retries still seem to give you the data.

More details for the 509 errors

21:07:17 | ( 1 / 1754) fetching game details for 140_game...
...
21:38:00 | (1320 / 1754) fetching game details for spellforce_platinum...
21:38:02 | request failed: 509 Server Error: Unknown for url
...
21:38:17 | failed to fetch https://www.gog.com/downloads/spellforce_platinum/5073

The 509 errors start with game 1320
Some of them fetch after a few retries, and some of them give a failed and are not in the manifest

22:07:23 | (1377 / 1754) fetching game details for stick_it_to_the_man...

From game 1377 (1 hour), the 509 errors disappear and there are no problems with the rest of the games, with one exception:

22:07:38 | (1389 / 1754) fetching game details for strike_suit_zero...
22:07:38 | request failed: 509 Server Error: Unknown for url