Yes, but they have never interfered with logging into GOG before and they still don't. I don't have any problems solving the ReCaptcha and logging in. It's just that I didn't get any ReCaptcha until a few days ago and now I always have to do one when logging in.

Thanks for troubleshooting this. I just wish it made sense. You can look at the commit
( https://github.com/Kalanyr/gogrepo/commit/27da409da0a0713360e16f04ebb255a7e2fd15eb )
and see the only change is the addition of a timeout when GOG is non-responsive for over 30 seconds. There's no change whatsoever to the actual download code unless a timeout occurs and that's a reset


I also can't remove the commit because the downloader can hang without it.

If anyone can spot anything in that commit that may cause slowdowns let me know.

Edit - does the slowdown persist if you let it go ? The other thing to try would be to see the HTTP_TIMEOUT to 120 instead of 30.

Could the problem be GOG is throttling large downloads due to increased traffic from the sale.

Then why would it happen only with certain versions of gogrepo?

I've run a full manifest update + download twice during the sales I think, and I don't recall seeing any such issues with download speeds, at least it has been able to max out my 10Mbps line pretty much constantly as far as I can tell. With my 4G mobile connection I've seen more variation with download speed, but I gather that is just its nature, it is not as stable as the fixed cable modem internet.

I think I downloaded probably something like 80GB of stuff in the first run, and maybe 40-50 GB in the second one. Not sure which gogrepo version exactly I am using, I did dowload a new version a couple of times now.

Some more tests this morning. Luckily I am working at home...
The slowdown persists. I let my first test run for 40 minutes untill I killed the process. Didn't have the nerves to wait for the server to close the line which happened when I was using 27da409d for the first time (don't have the log file anymore).
Tried changing the HTTP_TIMEOUT variable to 120, but couldn't notice any change.

Here are some excerpts from my log file:

08:55:26 | 7.9MB 0.7MB/s 1x 1954_alcatraz/1954alcatraz_artworks.zip
08:55:26 | 3.2MB 2.1MB/s 1x 1954_alcatraz/patch_1954_alcatraz_2.0.1.3.exe
08:55:26 | 1112.3MB 2.2MB/s 1x 1954_alcatraz/setup_1954_alcatraz_2.1.0.4.exe
08:55:26 | 199.1MB 5.6MB/s 1x 8th_wonder_of_the_world/setup_8th_wonder_2.1.0.9.exe
08:55:26 | 777.09GB remaining
08:55:27 | 7.5MB 0.5MB/s 1x 1954_alcatraz/1954alcatraz_artworks.zip
08:55:27 | 2.0MB 1.2MB/s 1x 1954_alcatraz/patch_1954_alcatraz_2.0.1.3.exe
08:55:27 | 1111.2MB 1.2MB/s 1x 1954_alcatraz/setup_1954_alcatraz_2.1.0.4.exe
08:55:27 | 196.9MB 2.2MB/s 1x 8th_wonder_of_the_world/setup_8th_wonder_2.1.0.9.exe
08:55:27 | 777.09GB remaining
08:55:28 | 7.0MB 0.4MB/s 1x 1954_alcatraz/1954alcatraz_artworks.zip
08:55:28 | 0.9MB 1.0MB/s 1x 1954_alcatraz/patch_1954_alcatraz_2.0.1.3.exe
08:55:28 | 1110.1MB 1.1MB/s 1x 1954_alcatraz/setup_1954_alcatraz_2.1.0.4.exe
08:55:28 | 194.4MB 2.5MB/s 1x 8th_wonder_of_the_world/setup_8th_wonder_2.1.0.9.exe

08:55:41 | 1097.6MB 1.1MB/s 1x 1954_alcatraz/setup_1954_alcatraz_2.1.0.4.exe
08:55:41 | 0.0MB 1.2MB/s 1x 8th_wonder_of_the_world/cultures_3_4_editor_docs.zip
08:55:41 | 172.6MB 1.1MB/s 1x 8th_wonder_of_the_world/setup_8th_wonder_2.1.0.9.exe
08:55:41 | 170.0MB 1.0MB/s 1x a_bird_story/setup_a_bird_story_1.0_(12441).exe
08:55:41 | 777.02GB remaining
08:55:41 | xml parsing error occurred trying to get md5 data for https://cdn-hw.gog.com/secure/a_golden_wake/pc/setup_a_golden_wake_2.0.0.4.exe.xml?ttl=1513842946&hw_l=56&hw_token=ab3466c468ff720a3c7b1d98651b8306&_token=xxx=hw
08:55:42 | 1096.6MB 1.0MB/s 1x 1954_alcatraz/setup_1954_alcatraz_2.1.0.4.exe
08:55:42 | 171.6MB 1.0MB/s 1x 8th_wonder_of_the_world/setup_8th_wonder_2.1.0.9.exe
08:55:42 | 168.9MB 1.0MB/s 1x a_bird_story/setup_a_bird_story_1.0_(12441).exe
08:55:42 | 379.5MB 0.9MB/s 1x a_golden_wake/setup_a_golden_wake_2.0.0.4.exe
08:55:42 | 777.02GB remaining
08:55:43 | 1095.7MB 0.9MB/s 1x 1954_alcatraz/setup_1954_alcatraz_2.1.0.4.exe
08:55:43 | 170.6MB 0.9MB/s 1x 8th_wonder_of_the_world/setup_8th_wonder_2.1.0.9.exe
08:55:43 | 168.1MB 0.9MB/s 1x a_bird_story/setup_a_bird_story_1.0_(12441).exe
08:55:43 | 378.6MB 0.9MB/s 1x a_golden_wake/setup_a_golden_wake_2.0.0.4.exe
08:55:43 | 777.02GB remaining

08:55:49 | 1091.3MB 0.7MB/s 1x 1954_alcatraz/setup_1954_alcatraz_2.1.0.4.exe
08:55:49 | 166.5MB 0.7MB/s 1x 8th_wonder_of_the_world/setup_8th_wonder_2.1.0.9.exe
08:55:49 | 163.2MB 0.7MB/s 1x a_bird_story/setup_a_bird_story_1.0_(12441).exe
08:55:49 | 373.7MB 0.7MB/s 1x a_golden_wake/setup_a_golden_wake_2.0.0.4.exe

08:55:56 | 1086.8MB 0.5MB/s 1x 1954_alcatraz/setup_1954_alcatraz_2.1.0.4.exe
08:55:56 | 162.1MB 0.5MB/s 1x 8th_wonder_of_the_world/setup_8th_wonder_2.1.0.9.exe
08:55:56 | 159.0MB 0.5MB/s 1x a_bird_story/setup_a_bird_story_1.0_(12441).exe
08:55:56 | 369.3MB 0.5MB/s 1x a_golden_wake/setup_a_golden_wake_2.0.0.4.exe

08:56:26 | 1074.5MB 0.3MB/s 1x 1954_alcatraz/setup_1954_alcatraz_2.1.0.4.exe
08:56:26 | 149.5MB 0.3MB/s 1x 8th_wonder_of_the_world/setup_8th_wonder_2.1.0.9.exe
08:56:26 | 146.3MB 0.3MB/s 1x a_bird_story/setup_a_bird_story_1.0_(12441).exe
08:56:26 | 356.3MB 0.3MB/s 1x a_golden_wake/setup_a_golden_wake_2.0.0.4.exe
08:56:26 | 776.93GB remaining

09:01:33 | 1013.3MB 0.1MB/s 1x 1954_alcatraz/setup_1954_alcatraz_2.1.0.4.exe
09:01:33 | 88.9MB 0.1MB/s 1x 8th_wonder_of_the_world/setup_8th_wonder_2.1.0.9.exe
09:01:33 | 84.9MB 0.1MB/s 1x a_bird_story/setup_a_bird_story_1.0_(12441).exe
09:01:33 | 294.9MB 0.2MB/s 1x a_golden_wake/setup_a_golden_wake_2.0.0.4.exe

09:29:41 | 867.4MB 0.0MB/s 1x 1954_alcatraz/setup_1954_alcatraz_2.1.0.4.exe
09:29:41 | 147.4MB 0.1MB/s 1x a_golden_wake/setup_a_golden_wake_2.0.0.4.exe
09:29:41 | 60.9MB 0.1MB/s 1x a_new_beginning/a_new_beginning_soundtrack.zip
09:29:41 | 3618.3MB 0.0MB/s 1x a_new_beginning/setup_a_new_beginning_final_cut_2.2.0.7.bin
09:29:41 | 776.12GB remaining

09:36:10 | 845.4MB 0.1MB/s 1x 1954_alcatraz/setup_1954_alcatraz_2.1.0.4.exe
09:36:10 | 125.0MB 0.1MB/s 1x a_golden_wake/setup_a_golden_wake_2.0.0.4.exe
09:36:10 | 38.9MB 0.1MB/s 1x a_new_beginning/a_new_beginning_soundtrack.zip
09:36:10 | 3596.2MB 0.1MB/s 1x a_new_beginning/setup_a_new_beginning_final_cut_2.2.0.7.bin
09:36:10 | 776.04GB remaining
09:36:11 | exiting...

My guess is that gog's server does not accept the login data and is closing the line to prevent a fraudulent download of my files. The loss of speed is too constant. But I do not know much about this matter.

Ok, I've updated the old manifest.
All went ok, except for:

- dreamfall_chapters_season_pass (didn't work even when retrying with a single ID)

request failed: 404 Client Error: Not Found for url: https://cdn-hw.gog.com/secureaaaa?(big code here)
failed to fetch https://www.gog.com/downloads/dreamfall_chapters_season_pass/77333

- 3 Shadowgate extras

shadowgate_retro_wallpapers.zip already exists in this game entry with same size/md5, skipping adding this file to the manifest
(etc)

Anyway, I'll try gogrepo.py download now

On the one hand YES this is great, but on the other...R.I.P. my HDD as I have 590 fuckin' games! xD

Uh.. I have some problems with gogrepo download:

- where's the destination optional path? Please add it again.
I'd like to keep gogrepo on the pc and use it on external drives.

- skipgames is deprecated? It was so convenient, can't you just keep it as a combo of the other 3 commands?

- if I chose os,lang,standalone already in the manifest, do I need to repeat them in the "download"?

DUDE Screw Dreamfall lost chapters... same here!

Mine in full list format that fail on manfest grab are:

dreamfall_chapters_season_pass
kyn
sublevel_zero
walking_dead_season_1_the
x3_terran_war_pack

But these always ALWAYS get screwed up for me :/

Destination should still be there gorepo download "Destination" -flags

It's still functional for now and works as you describe, the deprecated tag is merely a warning it may be removed in future.

Yes for os / lang assuming it's not Windows / English , no for installers which defaults to both for download and so will grab everything in the manifest.

The kyn / dreamfall errors are normal. It's missing md5s or files on GOG side. As are the duplicates (unless a rename happens) GOG lists certain files twice for some reason, it seems to be things that are both Extras and Part of Collectors editions / digital upgrade packs.

@RupertMurdock

Hmm. I notice you have a significant fraction of a terrabyte of games there, how much do you get through before it slows each time ? I haven't tested much over 200 GB myself (since I've never downloaded my entire collection from scratch).

Though I still don't know how the commit in question could be responsible, the one most likely to cause server issues are the last one your report fine.

I don't think it would be a login issue , detecting invalid data should close the connection and possibly ban you, not slow the connection, and not sending any data should result in the connection being forcibly closed after 30 seconds , that's what the commit you're having issues with does.

Neither have I... It is just for testing purposes.

I have checked my log files, and considering the moment when all concurrent downloads reach a maximum of 1 MB/s as the beginning of the slowdown, the drop occurs after 16-20 seconds and I can download between 70 and 100 MB.
I just had one instance where I downloaded only one large file and this download ran 1.05 minutes fine and I grabbed nearly 1 GB.
EDIT: Mea culpa. Not 1 GB only 180 MB. Wrote down the wrong numbers.

I test a bit more this evening. Maybe a process running in the background is interfering somehow. I will kill anything dispensable and try again.

What about you? You have the same troubles. Can you add additional information which could bring us forward? If it is just me I would say it is just my old machine which urgently needs a hardware upgrade.

Ah ok, in the help it was explained as:
gogrepo download -flags savedir
but that didn't work.

Ok, but that's the same problem anyway.
Can you please avoid deprecating it?

That's likely because you ended on an open ended flag like -ids or -lang or -os and it couldn't tell the directory from a parameter, it should still work in that format if you're last flag is closed.

I'm not going to remove it for no reason but if it becomes a hassle to maintain or causes the command set to become too bloated I'll consider removing it.

Is it possible to skip some files? (not independent games\IDs)

For example:
- setup_banished_32_1.0.7_(14938) -- (~150Mb, 32bit version)
- setup_dreamfall_chapters_32_5.7.4_(14247) -- (~9Gb, 32bit version)
- setup_dungeon_rats_32_1.0.6.0028_(14031) -- (~700Mb, 32bit version)
- setup_hard_west_32_1.5_(13043)-1.bin -- (~3.5Gb, 32bit version)
- setup_oxenfree_32_2.6.0_cs_hotfix_(17222) -- (~2Gb, 32bit version)
- setup_prison_architect_32_12b_(13214) -- (~350Mb, 32bit version)
- setup_prison_architect_32_13f_(17245)_(g) -- (~500Mb, Galaxy?)
- setup_sw_battlefront2_2.0.0.5-1 -- (~3.5Gb, old version still compatible with Xp & Vista)

Transfered my test folder from an external to an internal hdd, killed every superfluous process and made a quick test with commits 350b8b962 (=old) and 27da409d (=new).
As expected the same results: the old one downloads flawlessly the new one slows down.
But I noticed that the cpu load of python.exe is much higher running the new file.
The old one needs about 25% of my capacity at the start and stagnates at 4% during the download.
The new one is constantly higher than 30%.
Is this helpful in any way?

What else could I try out, any ideas, anyone?

If we can't find the culprit, I'll stick with the solution I already mentioned, I will use the new versions for updating and verifying and the old ones for downloading. Not a big deal.