Damn you gog! One thing you can currently do is change the delay between http get's..

Currently it's:
HTTP_FETCH_DELAY = 1 # in seconds

I would try 1.5 or 2. It will make your update longer, but it should avoid ban.

I have it set to 1 with 700+ games and I don't get banned... but who knows how their banning really works. It does take a long time, but that's because there's a fetch per game for the game data, and a fetch per file to get MD5 data. That's the killer.

I can add support for "resuming" updates... but ultimately I'd like to figure out a way to avoid ban.

I don't know of any way to "speed up update" without sacrificing fetching the MD5 info.

Any suggestions? :)

Also thanks for your feature suggestion list.

For varying definitions of "very wasteful":

The Blackwell Bundle: 4x147 MB = 441 MB wasted
Descent 1/2: 2x214 MB = 214 MB wasted
Heroes of Might and Magic V: 2x1361 MB = 1361 MB wasted
Wing Commander Prophecy/Secret Ops: 2x258 MB = 258 MB wasted

Personally, my preference is to keep them like GOG lists them, duplicates and all; hopefully in time they'll be separated and cleaned out. But I can see how someone might quail at HOMM5. ;)

Now I'll have to act like a total newbie: is there an tutorial how to install the needed dependencies (python and html5lib?) for Windows 7/64bit? Like, what exactly to download where, how to install them etc.?

I presume that from e.g. here https://www.python.org/downloads/ you are supposed to download and install Python 2.7.10, not 3.4.3?

I presume html5lib should be downloaded and installed from here: https://github.com/html5lib/html5lib-python but the installation part left me scratching my head. Do I go to Windows command prompt and just give there either "pip install html5lib" or "setup.py install" or what?

Figures I've used python only on Linux so far... For some reason I didn't have these problems figuring these out there, I already ran gogrepo on my Linux Mint PC to get md5sums for all my 977 GOG titles (that took around 3½ hours I think; seemed to work pretty good, sometimes the script couldn't obtain the data first time, but an automatic retry seemed to always work). If I understood right there was/is some issue with account with lots of games, but at least for me it succeeded to get all md5sums. I haven't tried to download them yet though, I need to do that on Windows.

It's odd I had to even ask anything like this, as I've occasionally done some light python development at work... (on Linux) :) But it's been awhile.


A bit related question: is there any estimation how much space total do all GOG installers take? Just trying to figure out at which ballpark. From my past experience and what installers I already have downloaded, I'd think a 2 terabyte HDD should be easily big enough for e.g. 977 GOG games, but then the size of GOG games have increased with lots of new games...

I thought I had already figured it out, but still some error. Done so far (on Windows 7/64bit):

1. Downloaded and installed python-2.7.10.msi from https://www.python.org/downloads/

2. From https://github.com/html5lib/html5lib-python, downloaded (as zip) html5lib-python-master.zip, uncompressed it, and in the command prompt gave "setup.py install", as per one of the messages above. At least this did something, lots of "copying" and "running" commands listed, there was one warning in the beginning: 3. From https://github.com/eddie3/gogrepo downloaded and uncompressed gogrepo-master.zip.

Giving the login command gives an error: So what am I missing?

Googling for it more:

http://stackoverflow.com/questions/17258003/cant-open-html5lib-in-python

So am I now missing "six"? Is that some additional prerequisite that is not currently mentioned, in addition to python 2.7.x and html5lib?

I see these instructions "$ pip install blaablaablaa", but where exactly am I supposed to give those "pip" commands? Windows command prompt obviously doesn't know such command, and doesn't seem to work in python either: What the heck is "pip"?

pip is an install manager. It's included in Python starting with version 2.7.9 and 3.4. You will have to run

import pip

from the python console, but using get-pip.py described in the link below pip being built-in sounds easier.

Thanks, import pip seems to work on python console (at least I don't get any error)... but I'm still lost. "pip install six" still doesn't work (nor "pip install html5lib"): It seems I am seriously now misunderstanding something how to use this all. I don't know why it was so easy to get to run on Linux (probably all needed dependencies were pretty much installed there by default), but I am constantly struggling on Windows to achieve the same.

Is there a tutorial somewhere what to do exactly, to get everything installed and ready on Windows? Does anyone actually have this working on Windows 7, or are people using it only on Linux?

Maybe I got it working, trial and error... So in Windows command prompt (running it as administrator just in case) in the folder where Python is installed, I had to do the following: After that head in command prompt where you have gogrepo.py, and it works:

C:\GOGRepo\gogrepo-master>gogrepo.py login
enter username:
enter password:
01:56:43 | attempting gog login as 'xxxx' ...
01:56:46 | login successful!

C:\GOGRepo\gogrepo-master>

PHEW!!!

EDIT: Forgot one important step in the beginning, downloading get-pip.py.

Ps. I ran yesterday into the problem that it couldn't get MD5sums for most of my games, skipping them after retry. Is this because of GOG blocking connection attempts after certain threshold from one user, or is the site just extraordinarily busy right now, due to sales? I think it was able to collect all md5sums a few days ago when I ran the scrtip on Linux, the summer sales was not on yet. I didn't fully check though if each and every md5sum came down, but that's the impression I got, ie. I didn't locate errors (several retries though).

Maybe I'll test this more after the summer sales, not causing too much burden...

Yeah GOG bans you if your "web request" rate to their server is too high. Nobody really knows what the threshold is. I'm sure summer sales generating high hits doesn't help either, although I think this is tracked by IP address (as a single user!)

However, in my script you can change the "HTTP_FETCH_DELAY" variable in the script to a higher number. This adds a delay (in seconds) any time a request() is made to GOG's servers... basically every web fetch request.

This of course adds a lot of time to how long the script takes to update and download... but if you run this late at night in a cronjob like I do, then how long it takes isn't really a big deal... and everything is usually done by the morning.

Look for this in the script:

# HTTP request settings
HTTP_FETCH_DELAY = 1 # in seconds

Perhaps try 1.5 or 2 seconds. Last night I tried 0.5 and got banned during MD5 as well.

Aside from the temp ban, sometimes requests() fail, that's why I added the auto-retry.. but this is normal. I get a few of these every run, but they always succeed after an auto retry or two.

-----------

For the record, this is what a GOG temp banning looks like:

02:04:46 | request failed: HTTP Error 503: Service Unavailable. will not retry.
02:04:46 | failed to fetch https://www.gog.com/downlink/master_of_orion/en1installer1. skipping...
02:04:47 | request failed: HTTP Error 503: Service Unavailable. will not retry.
02:04:47 | failed to fetch https://www.gog.com/downlink/master_of_orion/en3installer1. skipping...
02:04:48 | request failed: HTTP Error 503: Service Unavailable. will not retry.
02:04:48 | failed to fetch https://www.gog.com/downlink/master_of_orion/en3patch1. skipping...
02:04:49 | request failed: HTTP Error 503: Service Unavailable. will not retry.

Thanks for the tips.

What kind of connections does the script take (port etc.)? I didn't yet try it at my workplace (working from home today), but at least if I enable my work VPN, gogrepo.py connections apparently fail (and I presume it is the same at my workplace, corporate network). At my workplace, it goes like this:

+ Downloads from GOG.com with a web browser work.
+ The legacy GOG Downloader client works (log in and downloading GOG games with it).
+ EA Origin client works (which I actually found a bit surprising, as e.g. Steam doesn't).

- GOG Galaxy client doesn't work (can't log in)
- Steam client doesn't work (can't connect to Steam servers)
- gogrepo.py apparently doesn't work (can't connect?)

So something in the network (firewall, proxy, whatever) blocks the last three. However, I was hoping gogrepo.py would work fine, if it uses similar connections as when downloading with e.g. a web browser? I recall someone mentioning that's why GOG Downloader client works for me too, it uses simple http connections.

Do you know why that might be, with gogrepo.py? Should it work similarly if web browser downloads work?

Oh well, so much for the idea of downloading all my GOG games over some weekend at my work over a superfast connection. I guess I'll have to do it at my home over a measly 10Mbps cable modem connection. Luckily I don't have any monthly caps to worry about, so it just will mean more waiting. I'm kinda interested to know how long it will take to download almost 1000 GOG games with extras at my home, though. :)

woolymethodman YOU ROCK!

I do not know a thing about programming and such (can hardly program my coffee maker) so you are (all code writing guys and dolls) kind of super heroes to me.

Thanks for this great script! I just love it. Makes my library downloading a walk in the park...

+1

With some games (at least I noticed System Shock 2, and The Witcher 3), I get errors like this with gogrepo.py update: What does that mean in practice? What doesn't it find, the patches, or even the main game installers? Have you faced the same problem?

Ps. I changed the values a bit, http messages from 1s => 2s delay, retries after 10s, and 10 retries. Certain is certain. :)

Thanks! Glad you're enjoying it. I love having my DRM free collection at home and offline :)
Yeah I get the same for system shock. It's a "bug" in GOG's database.. they don't always have MD5 data for all files. I always try to fetch them, but if they don't exist you'll get 404 like that. You can just ignore those. It won't affect your ability to download the game itself, you just won't have the MD5 data for the particular file.

File size will still be checked in "verify", but there's not much I can do for the files that GOG does not give MD5 for (aside from starting a new community database, etc).

The script does nothing special.

These are the base URLs:

# GOG URLs
GOG_HOME_URL = r'http://www.gog.com'
GOG_AJAX_URL = r'https://www.gog.com/account/ajax'
GOG_LOGIN_URL = r'https://login.gog.com/login_check'

Could it be because https:// (port 443) instead of http:// (port 80) ?

I"m not sure they will work without being https... also the "download" links are taken right from data received from GOG. I don't know why gog downloader would work.. would have to wireshark or something and see what it's requesting.

Thanks for that script. Got it to run fine and it is downloading my games and extras just fine. Only problem seems to be that its not possible to download multiple languages. In my case english and german. I opened an issue with further info on github for that. Found this thread only now.