It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3861 posts)(3861 posts)
(3861 posts)
Pages:
1
50
93
94
95
96
97
150
200
250
258
This is my favourite topic
Posted March 02, 2018
avatar
Kalanyr: If they returned the correct error or had the approaprite metadata on their page headers which should indicate the desired timeout / and number of requests per hour respectively, I'd setup the script to respect it. But they don't, the error they return is the one for if an ISP is cutting off a site for having served too much data. I was actually going to rry and work out what GOG's per hour limit was by when the site started throwing 403 errors and then respect that, just before they added the delays.
avatar
kohlrak: I'm honestly surprised they haven't outright 403ed based on the user-agent string altogether. I get the impression that they don't like gogrepo, but don't really now how to get rid of it.
This thread is here on their forums and I know their community managers monitor general discussion at least, so they could always reach out to me (or wooly previously) , if they wanted me to limit it's requests per time period or even discontinue it (though I'd prefer the former to the latter of course).
Posted March 02, 2018
avatar
kohlrak: I'm honestly surprised they haven't outright 403ed based on the user-agent string altogether. I get the impression that they don't like gogrepo, but don't really now how to get rid of it.
avatar
Kalanyr: This thread is here on their forums and I know their community managers monitor general discussion at least, so they could always reach out to me (or wooly previously) , if they wanted me to limit it's requests per time period or even discontinue it (though I'd prefer the former to the latter of course).
Reaching out and communicating is something that i noticed they have a very difficult time doing.
Posted March 02, 2018
avatar
Lifthrasil: Is there a way to find out how big all my games are together without starting to download them? I would like to know how much disk space I need for a full backup.
python gogrepo.py login
- To get your login cookies, only needs to be done once, and if you lose your cookies or they become outdated.

python gogrepo.py update
- To get your game installer + extras details into a manifest file. This can take a long time, for me it takes for over 1400 games about 3½ hours or so, I think.
- By default this gets file details for English Windows versions. If you want also Mac and/or Linux versions and other languages besides English, use the approriate -os and -lang options, or I guess you can edit the script directly too so that they would be your default values always?

python gogrepo.py download -dryrun
- The dryrun option only simulates downloading without starting the actual downloads, so it will tell you how much there is to download in total.
Post edited March 02, 2018 by timppu
Posted March 02, 2018
avatar
Kalanyr: Correct,They show up on the game pages too, usually marked as coming soon , some of them have been coming soon for about 5 years at this point.
Ah yeah, I see it now! Dreamfall is still missing Journey's birth.
Posted March 02, 2018
avatar
Lifthrasil: Is there a way to find out how big all my games are together without starting to download them? I would like to know how much disk space I need for a full backup.
avatar
timppu: python gogrepo.py login
- To get your login cookies, only needs to be done once, and if you lose your cookies or they become outdated.

python gogrepo.py update
- To get your game installer + extras details into a manifest file. This can take a long time, for me it takes for over 1400 games about 3½ hours or so, I think.
- By default this gets file details for English Windows versions. If you want also Mac and/or Linux versions and other languages besides English, use the approriate -os and -lang options, or I guess you can edit the script directly too so that they would be your default values always?

python gogrepo.py download -dryrun
- The dryrun option only simulates downloading without starting the actual downloads, so it will tell you how much there is to download in total.
Great. Thank you!
Posted March 02, 2018
avatar
timppu: python gogrepo.py login
- To get your login cookies, only needs to be done once, and if you lose your cookies or they become outdated.

python gogrepo.py update
- To get your game installer + extras details into a manifest file. This can take a long time, for me it takes for over 1400 games about 3½ hours or so, I think.
- By default this gets file details for English Windows versions. If you want also Mac and/or Linux versions and other languages besides English, use the approriate -os and -lang options, or I guess you can edit the script directly too so that they would be your default values always?

python gogrepo.py download -dryrun
- The dryrun option only simulates downloading without starting the actual downloads, so it will tell you how much there is to download in total.
avatar
Lifthrasil: Great. Thank you!
-dryrun will be inaccurate if you have any interrupted download, as I didn't bother to precalculate how mich of "to be resumed" downloads are already downloaded, but that shoud be a relatively small amount amoount most of the time.
Posted March 03, 2018
Oh yes. Just while sort of on the topic of not being overly cruel to the GOG servers:

I don't recommend using a full update as your default. I recommend using -updateonly and -skipknown for your usual updates , and doing a quick check of (a) game(s) using -ids before installing it. Try limiting full updates to once a month or so (most of the time stuff picked up only by a full default is GOG doing some weird packaging shuffling that has no effect on anything user side and doing the check before installing a game will get any DLC/extras that may not have been flagged).

I may alter the script defaults to reflect this (so that it does a combination of updateonly / skipknown by default if a manifest exists). And add a -preinstallcheck that automatically does an update/clean/verify on a specified list of ids. To make this easier.
Posted March 03, 2018
Thank you for taking up the work on gogrepo Kalanyr!
Posted March 06, 2018
avatar
Kalanyr: Oh yes. Just while sort of on the topic of not being overly cruel to the GOG servers:

I don't recommend using a full update as your default. I recommend using -updateonly and -skipknown for your usual updates , and doing a quick check of (a) game(s) using -ids before installing it. Try limiting full updates to once a month or so (most of the time stuff picked up only by a full default is GOG doing some weird packaging shuffling that has no effect on anything user side and doing the check before installing a game will get any DLC/extras that may not have been flagged).
Yea that is a good best practice there.
Posted March 10, 2018
avatar
SirPrimalform: Thank you for taking up the work on gogrepo Kalanyr!
I second this!
Just switched to your repo, but you should mention:

pip install requests
pip install pyOpenSSL
p.s. wohooo 1000!

pps. that preallocation is bad? For example I removed the corrupted download of Kingdom come deliverance and wanted to redownload, it preallocates, then error because the download size does not match filesize from manifest, try again and all pass because the 'empty' files reallocated fit the file size.
This means, you have to know exactly where something went wrong because the script will create bogus files that match the files size/checksum before it starts downloading.
Post edited March 10, 2018 by disi
Posted March 10, 2018
Do we have an solid link we can use for Kalanyr's version we can refer to for the most up-to-date we can get? I'm having a hard time finding it in the thread.
Posted March 10, 2018
avatar
Starkrun: Do we have an solid link we can use for Kalanyr's version we can refer to for the most up-to-date we can get? I'm having a hard time finding it in the thread.
You can always get there from the github link in the first post by looking for fork top right and then click on his/her master version in the graph.
https://github.com/Kalanyr/gogrepo
Posted March 10, 2018
avatar
disi: You can always get there from the github link in the first post by looking for fork top right and then click on his/her master version in the graph.
https://github.com/Kalanyr/gogrepo
lol, that simple, thanks for the help.... I feel like an idiot now :)

annnnnd new bookmark added
Posted March 12, 2018
avatar
disi: pps. that preallocation is bad? For example I removed the corrupted download of Kingdom come deliverance and wanted to redownload, it preallocates, then error because the download size does not match filesize from manifest, try again and all pass because the 'empty' files reallocated fit the file size.
This means, you have to know exactly where something went wrong because the script will create bogus files that match the files size/checksum before it starts downloading.
That's why the preallocated files are in the !downloading folder where the download check and verify don't see them. They are only moved to the real final location if they pass a heuristic check (full checks would be prohibitively time expensive relative to the false positive rate (I've had 1) especially since the next verify will catch it).

The preallocation should also automatically trim/extend any partially preallocated files in !downloading if that data changed. The behaviour you're describing is a bug, though I'm not following what's happening from your description. If you could send me a copy of the appropriate log file I'll look into it.
Post edited March 12, 2018 by Kalanyr
Posted March 12, 2018
Like here on the 10/03/2018:

00:16:59 | loading local manifest...
00:16:59 | loading local resume manifest...
00:16:59 | fetching game product data (page 1)...
00:17:00 | fetching game product data (page 2 / 9)...
00:17:01 | fetching game product data (page 3 / 9)...
00:17:01 | fetching game product data (page 4 / 9)...
00:17:02 | scanning found "kingdom_come_deliverance" in product data!
00:17:02 | saving resume manifest...
00:17:02 | saved resume manifest
00:17:02 | (1 / 1) fetching game details for kingdom_come_deliverance...
00:17:17 | saving manifest...
00:17:20 | saved manifest
00:17:20 | saving resume manifest...
00:17:20 | saved resume manifest
00:17:20 | --
00:17:20 | total time: 0:00:20.831000
00:17:20 | exiting...
00:17:37 | loading local manifest...
00:17:38 | downloading games with id(s): {kingdom_come_deliverance}
00:17:38 | scanning manifest for renames...
00:17:38 | {kingdom_come_deliverance}
00:17:38 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170).exe
00:17:38 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-1.bin
00:17:38 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-2.bin
00:17:38 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-3.bin
00:17:38 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-4.bin
00:17:38 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-5.bin
00:17:38 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-6.bin
00:17:38 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-7.bin
00:17:38 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-8.bin
00:17:38 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-9.bin
00:17:38 | pass setup_kingdom_come_deliverance_treasures_of_the_past_dlc_1.3.1_g_(19170).exe
00:17:38 | ------------------------------------------------------------
00:17:39 | --
00:17:39 | total time: 0:00:01.429000
00:17:39 | exiting...
Here I deleted the false size files in the download and tried again (the installer files were corrupt on GOG side). It did not download anything, just preallocation and moving files around.

00:17:51 | loading local manifest...
00:17:51 | downloading games with id(s): {kingdom_come_deliverance}
00:17:51 | scanning manifest for renames...
00:17:51 | {kingdom_come_deliverance}
00:17:51 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170).exe
00:17:51 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-1.bin
00:17:51 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-2.bin
00:17:51 | download setup_kingdom_come_deliverance_1.3.1_g_(19170)-3.bin
00:17:51 | download setup_kingdom_come_deliverance_1.3.1_g_(19170)-4.bin
00:17:51 | download setup_kingdom_come_deliverance_1.3.1_g_(19170)-5.bin
00:17:51 | download setup_kingdom_come_deliverance_1.3.1_g_(19170)-6.bin
00:17:51 | download setup_kingdom_come_deliverance_1.3.1_g_(19170)-7.bin
00:17:51 | download setup_kingdom_come_deliverance_1.3.1_g_(19170)-8.bin
00:17:51 | download setup_kingdom_come_deliverance_1.3.1_g_(19170)-9.bin
00:17:51 | pass setup_kingdom_come_deliverance_treasures_of_the_past_dlc_1.3.1_g_(19170).exe
00:17:51 | ------------------------------------------------------------
00:17:51 | preallocating '2026849404' bytes for '.\!downloading\kingdom_come_deliverance\setup_kingdom_come_deliverance_1.3.1_g_(19170)-4.bin'
00:17:51 | preallocating '204086672' bytes for '.\!downloading\kingdom_come_deliverance\setup_kingdom_come_deliverance_1.3.1_g_(19170)-9.bin'
00:17:51 | preallocating '3534659584' bytes for '.\!downloading\kingdom_come_deliverance\setup_kingdom_come_deliverance_1.3.1_g_(19170)-3.bin'
00:17:51 | preallocating '284303957' bytes for '.\!downloading\kingdom_come_deliverance\setup_kingdom_come_deliverance_1.3.1_g_(19170)-7.bin'
00:17:53 | XML verification data size does not match manifest size for setup_kingdom_come_deliverance_1.3.1_g_(19170)-3.bin. manifest 3534659584, received 4294967294, skipping.
00:17:53 | not moving uncompleted download '.\!downloading\kingdom_come_deliverance\setup_kingdom_come_deliverance_1.3.1_g_(19170)-3.bin', success: False remaining bytes: 3534659584 / 3534659584
00:17:53 | preallocating '602451364' bytes for '.\!downloading\kingdom_come_deliverance\setup_kingdom_come_deliverance_1.3.1_g_(19170)-6.bin'
00:17:53 | XML verification data size does not match manifest size for setup_kingdom_come_deliverance_1.3.1_g_(19170)-7.bin. manifest 284303957, received 4294967294, skipping.
00:17:53 | not moving uncompleted download '.\!downloading\kingdom_come_deliverance\setup_kingdom_come_deliverance_1.3.1_g_(19170)-7.bin', success: False remaining bytes: 284303957 / 284303957
00:17:53 | XML verification data size does not match manifest size for setup_kingdom_come_deliverance_1.3.1_g_(19170)-9.bin. manifest 204086672, received 1854462901, skipping.
00:17:53 | preallocating '2300136198' bytes for '.\!downloading\kingdom_come_deliverance\setup_kingdom_come_deliverance_1.3.1_g_(19170)-5.bin'
00:17:53 | not moving uncompleted download '.\!downloading\kingdom_come_deliverance\setup_kingdom_come_deliverance_1.3.1_g_(19170)-9.bin', success: False remaining bytes: 204086672 / 204086672
00:17:53 | XML verification data size does not match manifest size for setup_kingdom_come_deliverance_1.3.1_g_(19170)-4.bin. manifest 2026849404, received 4294967294, skipping.
00:17:53 | preallocating '1461325754' bytes for '.\!downloading\kingdom_come_deliverance\setup_kingdom_come_deliverance_1.3.1_g_(19170)-8.bin'
00:17:53 | not moving uncompleted download '.\!downloading\kingdom_come_deliverance\setup_kingdom_come_deliverance_1.3.1_g_(19170)-4.bin', success: False remaining bytes: 2026849404 / 2026849404
00:17:54 | XML verification data size does not match manifest size for setup_kingdom_come_deliverance_1.3.1_g_(19170)-6.bin. manifest 602451364, received 4294967294, skipping.
00:17:54 | not moving uncompleted download '.\!downloading\kingdom_come_deliverance\setup_kingdom_come_deliverance_1.3.1_g_(19170)-6.bin', success: False remaining bytes: 602451364 / 602451364
00:17:54 | XML verification data size does not match manifest size for setup_kingdom_come_deliverance_1.3.1_g_(19170)-5.bin. manifest 2300136198, received 4294967294, skipping.
00:17:54 | not moving uncompleted download '.\!downloading\kingdom_come_deliverance\setup_kingdom_come_deliverance_1.3.1_g_(19170)-5.bin', success: False remaining bytes: 2300136198 / 2300136198
00:17:54 | XML verification data size does not match manifest size for setup_kingdom_come_deliverance_1.3.1_g_(19170)-8.bin. manifest 1461325754, received 4294967294, skipping.
00:17:54 | not moving uncompleted download '.\!downloading\kingdom_come_deliverance\setup_kingdom_come_deliverance_1.3.1_g_(19170)-8.bin', success: False remaining bytes: 1461325754 / 1461325754
00:17:54 | --
00:17:54 | total time: 0:00:03.465000
00:17:54 | exiting...
Then tried to download again:

00:18:10 | loading local manifest...
00:18:10 | downloading games with id(s): {kingdom_come_deliverance}
00:18:10 | scanning manifest for renames...
00:18:10 | {kingdom_come_deliverance}
00:18:10 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170).exe
00:18:10 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-1.bin
00:18:10 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-2.bin
00:18:10 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-3.bin
00:18:10 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-4.bin
00:18:10 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-5.bin
00:18:10 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-6.bin
00:18:10 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-7.bin
00:18:10 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-8.bin
00:18:10 | pass setup_kingdom_come_deliverance_1.3.1_g_(19170)-9.bin
00:18:10 | pass setup_kingdom_come_deliverance_treasures_of_the_past_dlc_1.3.1_g_(19170).exe
00:18:10 | ------------------------------------------------------------
00:18:11 | --
Post edited March 12, 2018 by disi
Pages:
1
50
93
94
95
96
97
150
200
250
258
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3861 posts)(3861 posts)
(3861 posts)