Kalanyr: On the topic of 509 bans
I tested it again with the latest gogrepo of yours, with these settings:
# HTTP request settings
HTTP_FETCH_DELAY = 1 # in seconds
HTTP_RETRY_DELAY = 30 # in seconds
HTTP_RETRY_COUNT = 30
HTTP_TIMEOUT = 30
So I increased both the delay and count a lot, as I don't want to hammer GOG servers constantly while being banned, and neither want to give up too easily.
The 509 errors started coming at almost exactly the same place as before, around game 1352 or so (when searching only for English Windows versions of games). These are the exact timestamps:
gogrepo.py update started:
13:44:22 | loading local manifest...
The 509 error comes the first time (after about 40 minutes from the start):
14:24:23 | (1352 / 1408) fetching game details for wing_commander_3_heart_of_the_tiger...
14:24:26 | request failed: 509 Server Error
The first game comes without errors:
14:44:49 | (1359 / 1408) fetching game details for wing_commander_secret_ops...
As said, the games during the ban do seem to get the data too, but only after several retries. For some one retry seemed to be enough (waiting for 30 seconds), while some needed 12 retries (a 30 second wait between each).
Since I don't know if this depends e.g. on max requests per hour, or is it based on time (constant requests for 40 minutes = ban?), would it make sense to add two new parameters to the script, something like:
HTTP_COOLDOWN_AFTER = 1300 # number of games, 0 = never
HTTP_COOLDOWN_PAUSE = 300 # in seconds
or something like that, user-configurable parameters for after how many games there is a cooldown period when no more http requests are sent, and how long that pause should be?
I don't know if this is wildly different for different people based on e.g. what versions they are trying to get, like if someone wants also Linux and Mac versions and several language versions, will they hit the limit much sooner (e.g. after 700 games)? Everyone would have to come up with values that suit their needs, and I guess the cooldown period time just need to be tested, ie. after how long wait you've evaded the incoming ban?
I guess people with only a few hundred games won't even see this issue then.
Or so you have some other idea how this should be handled?
EDIT: Looking at the timestamps, it seems apparent that there is a limit of http requests per hour (in both tests 509 errors were not received anymore exactly after one hour had passed since starting the gogrepo update). So the "cooldown pause" period should be set so that it won't continue until one hour has passed since the start of the update run.
If that was the case, then maybe there should be only one parameter ("http cooldown after"), after which gogrepo stops until one hour has passed since the start of the update run? That way the script would limit the amount of http requests per hour.
What I still don't understand is that GOG doesn't seem to block you completely during the ban, as several retries still seem to give you the data.