It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3861 posts)(3861 posts)
(3861 posts)
Pages:
1
50
78
79
80
81
82
150
200
250
258
This is my favourite topic
Posted December 06, 2017
avatar
Kalanyr: I realise you may not want to do this and it's fine if you don't , but if you email ( ) a copy of an exported copy of your cookies.txt from your Firefox build (preferably from a clean session with only a GOG login) , I'll make sure it's compatible and I'll delete it as soon as I've seen the format. Otherwise I'll upload another test build soon and you can let me know if it works.
avatar
tinyE: TAKE MY ADVICE and remove your email from that post.
Can I ask why ?
My email has been publicly available for years due to various work and personal reasons, is trivially guessable if you know my Github username and has a strong password for anything remotely important , as well as 2FA for anything very important.
Posted December 06, 2017
avatar
tinyE: TAKE MY ADVICE and remove your email from that post.
avatar
Kalanyr: Can I ask why ?
My email has been publicly available for years due to various work and personal reasons, is trivially guessable if you know my Github username and has a strong password for anything remotely important , as well as 2FA for anything very important.
There are people in here, typically non regs scammers/trolls, who get their kicks doxing people and making their life hell. I had a problem with one just a few weeks ago.

It's up to you, but if the wrong person in here gets that address you might start getting some pretty obscene stuff and possibly spammed into oblivion, not to mention cleverly disguised viruses.

ALSO, it violates the Forum COC but but I don't think anyone is going to get on you for that.
Post edited December 06, 2017 by tinyE
Posted December 06, 2017
EITerprise and phaolo

Could you each test this please: rename your gog-cookies.dat to something else and place your cookies.txt named cookies.txt in the same directory as gogrepo.py and do an update or rename a small game/extra file and do a download

https://www.dropbox.com/s/a89ewreqo05pdef/gogrepo.py?dl=0

If it works for both of you, I'll mark this one off the list.

ETA - To be clear here it needs to be a clean cookies.txt not one you tried to use as gog-cookies.dat already, unfortunately the previous import system was destructive.
Post edited December 06, 2017 by Kalanyr
Posted December 06, 2017
avatar
Kalanyr: I would recommend keeping the manifest file rather than nuking it for no reason yes, at this point in time there is literally no reason to nuke it (it automatically cleans up extra entries etc).
avatar
timppu: Still about this: what are the benefits of keeping the old manifest file (besides this new partial verification)? Does it handle it differently if e.g. a game directory has changed while the files themselves are still the same?

When getting a whole new manifest file, I guess it simply means that "download" will download all the game files again (just because the game's directory name had changed), and "clean" will consider the files under the old directory obsolete and move them away.

Would keeping the old manifest file handle that better, like just renaming the directory name, without downloading anything all over again? Then again this issue doesn't happen often, but I do remember back when GOG kept constantly changing The Witcher 3 title name, and I ended up redownloading all the TW3 installer and extras files several times... :)
I've thought about this a bit further and I'm going to add the ability to use Python regular expressions in the -ids and -skipids arguments , as well as add a tag to make the clearing of the previous verification flag on missing files optional. That should let you manage multiple storage locations from a single small folder on your computer containing only gogrepo, gog-cookies.dat , and the manifest and resume manifest

I'll add some exampes for common stuff like a* or all games starting with a-n .
Post edited December 06, 2017 by Kalanyr
Posted December 06, 2017
So I can report that your solution works with Python 2 but only if I move to the gogrep folder in Powershell. Otherwise it can't find the "cookies.txt".
(something like "py -2 D:\gogrepo\gogrepo.py )

With Python 3 it gives the following error:

Traceback (most recent call last):
File "gogrepo.py", line 2299, in <module>
main(process_argv(sys.argv))
File "gogrepo.py", line 2211, in main
cmd_update(args.os, args.lang, args.skipknown, args.updateonly, args.ids, args.skipids,args.skiphidden,args.installers,args.resumemode,args.strictverify)
File "gogrepo.py", line 1011, in cmd_update
updateSession = makeGOGSession()
File "gogrepo.py", line 994, in makeGOGSession
load_cookies()
File "gogrepo.py", line 309, in load_cookies
line = line.replace("#HttpOnly_","")
TypeError: a bytes-like object is required, not 'str'
Post edited December 06, 2017 by ElTerprise
Posted December 06, 2017
avatar
ElTerprise: So I can report that your solution works with Python 2 but only if I move to the gogrep folder in Powershell. Otherwise it can't find the "cookies.txt".
(something like "py -2 D:\gogrepo\gogrepo.py )

With Python 3 it gives the following error:

Traceback (most recent call last):
File "gogrepo.py", line 2299, in <module>
main(process_argv(sys.argv))
File "gogrepo.py", line 2211, in main
cmd_update(args.os, args.lang, args.skipknown, args.updateonly, args.ids, args.skipids,args.skiphidden,args.installers,args.resumemode,args.strictverify)
File "gogrepo.py", line 1011, in cmd_update
updateSession = makeGOGSession()
File "gogrepo.py", line 994, in makeGOGSession
load_cookies()
File "gogrepo.py", line 309, in load_cookies
line = line.replace("#HttpOnly_","")
TypeError: a bytes-like object is required, not 'str'
avatar
ElTerprise:
.
The first is the case for loading the manifest as well anyway isn't it ? I'll double check the loading method and make sure they are the same.

Sigh. Once more Python 3 fails to.handle Strings as Unicode by default transparently. I'll beat it with a stick and upload a new test version in a minute.
Posted December 06, 2017
avatar
Kalanyr: Sigh. Once more Python 3 fails to.handle Strings as Unicode by default transparently. I'll beat it with a stick and upload a new test version in a minute.
Isn't this the usual problem with open() treating every file as binary on Windows?
Posted December 07, 2017
@EITerprise

I've updated the test version again, if you could redownload it and try it out, it should work in python 2 and python 3 now. Hopefully the functioning of cookies.txt, gog-cookies.dat , gog-manifest.dat and gog-resume-manifest.dat are all consistent (I think you should need to be in the folder or be using an invocation method that natively treats the folder something is called from as it's working directory).
Post edited December 07, 2017 by Kalanyr
Posted December 07, 2017
avatar
Kalanyr: @EITerprise

I've updated the test version again, if you could redownload it and try it out, it should work in python 2 and python 3 now. Hopefully the functioning of cookies.txt, gog-cookies.dat , gog-manifest.dat and gog-resume-manifest.dat are all consistent (I think you should need to be in the folder or be using an invocation method that natively treats the folder something is called from as it's working directory).
I've tested both Python 2 and 3 and it works in both now. Thank you very much for fixing it!
Posted December 07, 2017
Just putting a reminder here for myself to add an import_cookies command to automatically handle deleting gog-cookie.dat for ease of use and an appropriate error message about missing cookies.txt and how to get it.
Post edited December 07, 2017 by Kalanyr
Posted December 07, 2017
avatar
Kalanyr: email me a copy of an exported copy of your cookies.txt from your Firefox build (preferably from a clean session with only a GOG login)
Your chat seems to be blocked, I sent you a friend request.
I don't have any cookies.txt at the moment.
But I haven't tried your script yet, maybe in the weekend.
(I'll probably need some info)
Post edited December 07, 2017 by phaolo
Posted December 07, 2017
avatar
Kalanyr: The former happens when GOG has the items in question double listed. It's just a matter of keeping the manifest tidy.

Thats probably not something I can do anything about, most NAS run Linux behind the scenes which means trying to do Preallocation from Windows is not something I'm up for. Though I thought I was catching that exception, hopefully it's just diagnostic logging I forgot about.
avatar
ikrananka: I'm curious to know if others are also seeing the same "already exists" messages. It just seems odd as I've never seen them before.

As I'm downloading directly to an ext4 multi-drive NAS in RAID mode, pre-allocation is really not something I'm looking for. The games in question downloaded and verified fine. I'll let you know if these kind of errors reoccur so you can correct your error exceptions as necessary.
I get the "already exists" message on some games when updating with multiple languages, such as:
[command]
python3 gogrepo.py update -os windows linux mac -lang en cn it cz ar gk es ru nl pt no tr ro pl fr bl de jp hu br fi da ko sv sk sb
[/command]

When using the original gogrepo.py, I seem to recall those same games had one executable file for multiple languages.
I also seem to recall the original manifest had multiple entries for the different languages, all listing the same file with the same md5.
Posted December 07, 2017
I am getting these errors:

fetching game details for divine_divinity...
00:20:29 | request failed: 404 Client Error: Not Found for url: https://cdn.gog.com/secure/divine_divinity/mac/german/divine_divinity_german_2.0.0.10.dmg.xml?057dec49d4bd4f1d64d002adbc13b9b272749de6d16e59845f04d678603f7da6ebaa26499d2418d6fa748f425141ce1d6ede9fd92445d17608fcb52588c5fc9ca76cf00a3b71e7dbbfb177b0759a45b9b64f2f941d8afb433823037ddfe2596d358004afc08b6902935e3e9c8754515208564b4bad6cf6c662d7a3cee9c78d697719bfb9654e97. will not retry.
00:20:29 | no md5 data found for divine_divinity_german_2.0.0.10.dmg

fetching game details for two_worlds...
00:30:34 | request failed: 404 Client Error: Not Found for url: https://cdn.gog.com/secure/two_worlds/pc/spanish/setup_two_worlds_epic_edition_spanish_2.2.0.23.exe.xml?057dd507a4a75b23ad7858e907590d3219dbf4d71d4fbe2faec8b805b07891aac7a69e03890921bf262ea1b50d26a531d2d506485ff30a240f0a4785d50bdfe1470fd4a1e046b96d2c63f40ca07a35b0c71cf5516f99408b02ce8e3497d9a8957ba705df9c15810a2f4b1ba49716f6419ab0a52d1d215cb9aa8439f388a144f4450f459396d0e2f5a3b38a6d041fda1db624b56e. will not retry.
00:30:34 | no md5 data found for setup_two_worlds_epic_edition_spanish_2.2.0.23.exe
00:30:34 | request failed: 404 Client Error: Not Found for url: https://cdn.gog.com/secure/two_worlds/pc/spanish/setup_two_worlds_epic_edition_spanish_2.2.0.23-1.bin.xml?057dec408e8e401f3b1e2ac0360bf9b217e9f238874d625c3a00dd033ccc5de3e0bb18ffbfc41e64b259466a6384e8a9f2091ff7bc626b4e74836a22c89270ff6726a51667a40e76d9ea8d5a10bb92b5cc9dae7c68526596012ce9b4e29aba9191ff80f3b03fb49628ec1676d8b95f0a3c1dd43c8ec6b45f4e1a05b667085fe1ddc209b43994fbb9d4a3c384fb390773b0079f. will not retry.
00:30:34 | no md5 data found for setup_two_worlds_epic_edition_spanish_2.2.0.23-1.bin
Posted December 07, 2017
avatar
solar_dome: I am getting these errors:

fetching game details for divine_divinity...
00:20:29 | request failed: 404 Client Error: Not Found for url: https://cdn.gog.com/secure/divine_divinity/mac/german/divine_divinity_german_2.0.0.10.dmg.xml?057dec49d4bd4f1d64d002adbc13b9b272749de6d16e59845f04d678603f7da6ebaa26499d2418d6fa748f425141ce1d6ede9fd92445d17608fcb52588c5fc9ca76cf00a3b71e7dbbfb177b0759a45b9b64f2f941d8afb433823037ddfe2596d358004afc08b6902935e3e9c8754515208564b4bad6cf6c662d7a3cee9c78d697719bfb9654e97. will not retry.
00:20:29 | no md5 data found for divine_divinity_german_2.0.0.10.dmg

fetching game details for two_worlds...
00:30:34 | request failed: 404 Client Error: Not Found for url: https://cdn.gog.com/secure/two_worlds/pc/spanish/setup_two_worlds_epic_edition_spanish_2.2.0.23.exe.xml?057dd507a4a75b23ad7858e907590d3219dbf4d71d4fbe2faec8b805b07891aac7a69e03890921bf262ea1b50d26a531d2d506485ff30a240f0a4785d50bdfe1470fd4a1e046b96d2c63f40ca07a35b0c71cf5516f99408b02ce8e3497d9a8957ba705df9c15810a2f4b1ba49716f6419ab0a52d1d215cb9aa8439f388a144f4450f459396d0e2f5a3b38a6d041fda1db624b56e. will not retry.
00:30:34 | no md5 data found for setup_two_worlds_epic_edition_spanish_2.2.0.23.exe
00:30:34 | request failed: 404 Client Error: Not Found for url: https://cdn.gog.com/secure/two_worlds/pc/spanish/setup_two_worlds_epic_edition_spanish_2.2.0.23-1.bin.xml?057dec408e8e401f3b1e2ac0360bf9b217e9f238874d625c3a00dd033ccc5de3e0bb18ffbfc41e64b259466a6384e8a9f2091ff7bc626b4e74836a22c89270ff6726a51667a40e76d9ea8d5a10bb92b5cc9dae7c68526596012ce9b4e29aba9191ff80f3b03fb49628ec1676d8b95f0a3c1dd43c8ec6b45f4e1a05b667085fe1ddc209b43994fbb9d4a3c384fb390773b0079f. will not retry.
00:30:34 | no md5 data found for setup_two_worlds_epic_edition_spanish_2.2.0.23-1.bin
That's normal on games with no md5 checksums... This version just lists the web address that is missing.
Posted December 07, 2017
avatar
ikrananka: I'm curious to know if others are also seeing the same "already exists" messages. It just seems odd as I've never seen them before.

As I'm downloading directly to an ext4 multi-drive NAS in RAID mode, pre-allocation is really not something I'm looking for. The games in question downloaded and verified fine. I'll let you know if these kind of errors reoccur so you can correct your error exceptions as necessary.
avatar
solar_dome: I get the "already exists" message on some games when updating with multiple languages, such as:
[command]
python3 gogrepo.py update -os windows linux mac -lang en cn it cz ar gk es ru nl pt no tr ro pl fr bl de jp hu br fi da ko sv sk sb
[/command]

When using the original gogrepo.py, I seem to recall those same games had one executable file for multiple languages.
I also seem to recall the original manifest had multiple entries for the different languages, all listing the same file with the same md5.
Are there any cases where the language versions are different but have the same name and get automatically renamed ? Rather than not being added.

I'm considering changing the way update works and it won't work if there are.
Pages:
1
50
78
79
80
81
82
150
200
250
258
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3861 posts)(3861 posts)
(3861 posts)