It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3896 posts)(3896 posts)
(3896 posts)
Pages:
1
50
77
78
79
80
81
150
200
250
260
This is my favourite topic
Posted December 06, 2017
avatar
ElTerprise: Well I'm currently not able to use gogrepo (using Kalanyr's fork) as the reCaptcha prevents me from logging in. Unfortunately I forgot to save the cookie when I reinstalled Windows a few days ago.

I've tried several things:
- waiting and retrying later
- logging out and clearing all cookies in my browser to trigger the reCaptcha there
- disabling 2FA

No matter what I do, I get a the reCaptcha notification in gogrepo.

The old workaround with exporting cookies from the browser also fails with the following error:

13:13:48 | loading local manifest...
13:13:48 | fatal...
Traceback (most recent call last):
File "gogrepo.py", line 2286, in <module>
main(process_argv(sys.argv))
File "gogrepo.py", line 2198, in main
cmd_update(args.os, args.lang, args.skipknown, args.updateonly, args.ids, args.skipids,args.skiphidden,args.installers,args.resumemode,args.strictverify)
File "gogrepo.py", line 998, in cmd_update
updateSession = makeGOGSession()
File "gogrepo.py", line 981, in makeGOGSession
load_cookies()
File "gogrepo.py", line 302, in load_cookies
tmp_jar.load()
File "(...)\Python\Python36\lib\http\cookiejar.py", line 1784, in load
self._really_load(f, filename, ignore_discard, ignore_expires)
File "(...)\Python\Python36\lib\http\cookiejar.py", line 2004, in _really_load
if not self.magic_re.search(magic):
AttributeError: 'str' object has no attribute 'search'
avatar
ElTerprise: What am I doing wrong? Or am I just out of luck at the moment?
This is caused by a work around wooly did to allow loading the cookie unfortunately Python 3 changes what wa a string in Python 2 into a compiled regular expression. Gimme a bit and I'll give you a quick fix.
Posted December 06, 2017
(In regards to the above post only)

Give this a try if you would:
https://www.dropbox.com/s/a89ewreqo05pdef/gogrepo.py?dl=0
Posted December 06, 2017
avatar
Kalanyr: The former happens when GOG has the items in question double listed. It's just a matter of keeping the manifest tidy.

Thats probably not something I can do anything about, most NAS run Linux behind the scenes which means trying to do Preallocation from Windows is not something I'm up for. Though I thought I was catching that exception, hopefully it's just diagnostic logging I forgot about.
I'm curious to know if others are also seeing the same "already exists" messages. It just seems odd as I've never seen them before.

As I'm downloading directly to an ext4 multi-drive NAS in RAID mode, pre-allocation is really not something I'm looking for. The games in question downloaded and verified fine. I'll let you know if these kind of errors reoccur so you can correct your error exceptions as necessary.
Posted December 06, 2017
avatar
Kalanyr: I would recommend keeping the manifest file rather than nuking it for no reason yes, at this point in time there is literally no reason to nuke it (it automatically cleans up extra entries etc).
avatar
timppu: Still about this: what are the benefits of keeping the old manifest file (besides this new partial verification)? Does it handle it differently if e.g. a game directory has changed while the files themselves are still the same?

When getting a whole new manifest file, I guess it simply means that "download" will download all the game files again (just because the game's directory name had changed), and "clean" will consider the files under the old directory obsolete and move them away.

Would keeping the old manifest file handle that better, like just renaming the directory name, without downloading anything all over again? Then again this issue doesn't happen often, but I do remember back when GOG kept constantly changing The Witcher 3 title name, and I ended up redownloading all the TW3 installer and extras files several times... :)
Yes, it will handle game / file name changes automatically, I added that with the big update, well unless you do something profoundly weird like update your manifest and then dont do a download, verify or clean before the next time you update after GOG changes the name again.


avatar
Kalanyr: The former happens when GOG has the items in question double listed. It's just a matter of keeping the manifest tidy.

Thats probably not something I can do anything about, most NAS run Linux behind the scenes which means trying to do Preallocation from Windows is not something I'm up for. Though I thought I was catching that exception, hopefully it's just diagnostic logging I forgot about.
avatar
ikrananka: I'm curious to know if others are also seeing the same "already exists" messages. It just seems odd as I've never seen them before.

As I'm downloading directly to an ext4 multi-drive NAS in RAID mode, pre-allocation is really not something I'm looking for. The games in question downloaded and verified fine. I'll let you know if these kind of errors reoccur so you can correct your error exceptions as necessary.
Yeah, I am. It's new to my fork since it's an error on GOG's side that I only found out about when they were uploading 2 different installers with the same file name and different md5s at one point , for extras it was unnoticeable since they'd just get clobbered and whichever was the latest one would pass the check because the extras with the same name were always the exact same file. It's harmless and just means you're manifest won't have 2 entries for exactly the same item even if GOG does for some reason (if the items are different it automatically renames one).

I did a quick check , and yeah that's just diagnostic logging, I'll leave that there for now due to other potential errors in that system. It being ext4 explains why Windows couldn't get what it considers a valid filehandle.
Post edited December 06, 2017 by Kalanyr
Posted December 06, 2017
avatar
Kalanyr: (In regards to the above post only)

Give this a try if you would:
https://www.dropbox.com/s/a89ewreqo05pdef/gogrepo.py?dl=0
Well without exported cookies, I'm still getting a reCaptcha notification - with exported cookies, it fails to load the cookies:
16:13:45 | loading local manifest...
16:13:45 | failed to load cookies, did you login first?
Should I try running it with Python2 as an alternative? I'm not really using Python for anything else...
Posted December 06, 2017
avatar
Kalanyr: (In regards to the above post only)

Give this a try if you would:
https://www.dropbox.com/s/a89ewreqo05pdef/gogrepo.py?dl=0
avatar
ElTerprise: Well without exported cookies, I'm still getting a reCaptcha notification - with exported cookies, it fails to load the cookies:

16:13:45 | loading local manifest...
16:13:45 | failed to load cookies, did you login first?
avatar
ElTerprise: Should I try running it with Python2 as an alternative? I'm not really using Python for anything else...
You can run Python2 / Python 3 together on Windows. So if you'd check if the cookie import works on Python 2 that would be great. I'm 99% sure that I fixed the difference though so I'd be surprised if it does unfortunately but if it works at least I can try and work out what's going wrong.

If you install both you can just use py -2 to use python2 and py -3 to use python3 .
Posted December 06, 2017
avatar
Kalanyr: You can run Python2 / Python 3 together on Windows. So if you'd check if the cookie import works on Python 2 that would be great. I'm 99% sure that I fixed the difference though so I'd be surprised if it does unfortunately but if it works at least I can try and work out what's going wrong.

If you install both you can just use py -2 to use python2 and py -3 to use python3 .
Still the same. Both with a cookies.txt and a gog-cookies.dat. Even tried with both a recent and an older cookie file...
Posted December 06, 2017
avatar
Kalanyr: You can run Python2 / Python 3 together on Windows. So if you'd check if the cookie import works on Python 2 that would be great. I'm 99% sure that I fixed the difference though so I'd be surprised if it does unfortunately but if it works at least I can try and work out what's going wrong.

If you install both you can just use py -2 to use python2 and py -3 to use python3 .
avatar
ElTerprise: Still the same. Both with a cookies.txt and a gog-cookies.dat. Even tried with both a recent and an older cookie file...
Yes, I'm having a look now and the function that clearly worked before is missing some rather important data this time. I'll poke at it for another 15 minutes or so and if I have no joy I'll have to sleep and move it to a tomorrow project.
Posted December 06, 2017
avatar
Kalanyr: Yes, I'm having a look now and the function that clearly worked before is missing some rather important data this time. I'll poke at it for another 15 minutes or so and if I have no joy I'll have to sleep and move it to a tomorrow project.
Okay. Anyway thank you for trying to help and fix it - there's no hurry - would be nice to have it working again at some point though ;)
Posted December 06, 2017
avatar
ikrananka: I'm curious to know if others are also seeing the same "already exists" messages. It just seems odd as I've never seen them before.

As I'm downloading directly to an ext4 multi-drive NAS in RAID mode, pre-allocation is really not something I'm looking for. The games in question downloaded and verified fine. I'll let you know if these kind of errors reoccur so you can correct your error exceptions as necessary.
avatar
Kalanyr: Yeah, I am. It's new to my fork since it's an error on GOG's side that I only found out about when they were uploading 2 different installers with the same file name and different md5s at one point , for extras it was unnoticeable since they'd just get clobbered and whichever was the latest one would pass the check because the extras with the same name were always the exact same file. It's harmless and just means you're manifest won't have 2 entries for exactly the same item even if GOG does for some reason (if the items are different it automatically renames one).

I did a quick check , and yeah that's just diagnostic logging, I'll leave that there for now due to other potential errors in that system. It being ext4 explains why Windows couldn't get what it considers a valid filehandle.
Cool - thanks for the clarifications. Thank you for all of your work on this, it really is most appreciated.
Post edited December 06, 2017 by ikrananka
Posted December 06, 2017
avatar
Kalanyr: Yes, I'm having a look now and the function that clearly worked before is missing some rather important data this time. I'll poke at it for another 15 minutes or so and if I have no joy I'll have to sleep and move it to a tomorrow project.
avatar
ElTerprise: Okay. Anyway thank you for trying to help and fix it - there's no hurry - would be nice to have it working again at some point though ;)
Well the good news is I've identified the problem:
The only working cookie exporter on WIndows for the latest version of Firefox uses a different format for exporting cookies than was originally expected and/or the Cookie importer in python is less relaxed now. I massaged the cookie.txt file a bit and managed to get it to import. So tomorrow I'l check if there's a way to loosen python's handling and if not just programmatically axe the bits that are causing problems.
Posted December 06, 2017
avatar
Kalanyr: Well the good news is I've identified the problem:
The only working cookie exporter on WIndows for the latest version of Firefox uses a different format for exporting cookies than was originally expected and/or the Cookie importer in python is less relaxed now. I massaged the cookie.txt file a bit and managed to get it to import. So tomorrow I'l check if there's a way to loosen python's handling and if not just programmatically axe the bits that are causing problems.
Oh that's good to hear.
Posted December 06, 2017
avatar
Kalanyr: latest version of Firefox uses a different format for exporting cookies
Uh, but please keep the compatibility with older Firefox versions.
I stopped at v54.01 because I didn't want to lose my legacy addons.
Posted December 06, 2017
avatar
Kalanyr: latest version of Firefox uses a different format for exporting cookies
avatar
phaolo: Uh, but please keep the compatibility with older Firefox versions.
I stopped at v54.01 because I didn't want to lose my legacy addons.
I realise you may not want to do this and it's fine if you don't , but if you email me a copy of an exported copy of your cookies.txt from your Firefox build (preferably from a clean session with only a GOG login) , I'll make sure it's compatible and I'll delete it as soon as I've seen the format. Otherwise I'll upload another test build soon and you can let me know if it works.

DM me for my email
Post edited December 06, 2017 by Kalanyr
Posted December 06, 2017
avatar
phaolo: Uh, but please keep the compatibility with older Firefox versions.
I stopped at v54.01 because I didn't want to lose my legacy addons.
avatar
Kalanyr: I realise you may not want to do this and it's fine if you don't , but if you email ( ) a copy of an exported copy of your cookies.txt from your Firefox build (preferably from a clean session with only a GOG login) , I'll make sure it's compatible and I'll delete it as soon as I've seen the format. Otherwise I'll upload another test build soon and you can let me know if it works.
TAKE MY ADVICE and remove your email from that post.
Pages:
1
50
77
78
79
80
81
150
200
250
260
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3896 posts)(3896 posts)
(3896 posts)