It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4078 posts)(4078 posts)
(4078 posts)
Pages:
1
50
100
150
190
191
192
193
194
250
272
This is my favourite topic
Posted February 15, 2022
Yep, same here since today with the files from https://github.com/Kalanyr/gogrepoc , but it seems that every entry in my library has this problem now...

I did redownload the gogrepoc but no changes of the situation.

I use Python 3.9.
Post edited February 15, 2022 by Genericjohndoe
Posted February 15, 2022
I hope this is just a server side problem with gog.com.
Yesterday everthing was fine and now multiple users have the same problem...
Posted February 15, 2022
avatar
Genericjohndoe: I hope this is just a server side problem with gog.com.
Yesterday everthing was fine and now multiple users have the same problem...
It's not necessarily a problem in the first place.
If the data format has been changed, the script has to be adapted in order to read it correctly.
Posted February 15, 2022
avatar
Genericjohndoe: I hope this is just a server side problem with gog.com.
Yesterday everthing was fine and now multiple users have the same problem...
avatar
neumi5694: It's not necessarily a problem in the first place.
If the data format has been changed, the script has to be adapted in order to read it correctly.
This is also affecting lgogdownloader, so it's not an issue with gogrepo. Something changed on GOG's end.
Posted February 16, 2022
avatar
neumi5694: It's not necessarily a problem in the first place.
If the data format has been changed, the script has to be adapted in order to read it correctly.
avatar
racofer: This is also affecting lgogdownloader, so it's not an issue with gogrepo. Something changed on GOG's end.
I just did some digging. The URLs to the CDN have changed.

So the URL manipulation to get the .xml with the file information instead of the actual file has to be rewritten. In my downloader it only took changing 2 lines of code so that everything was back to normal. So I guess gogrepo will also be updated pretty quickly.

However, some XML files also seem to be broken at the moment. For example the XML for /downloads/arma_gold_edition/en1installer3 is empty. It seems GOG also needs to do some fixing since their changes broke something.
Posted February 16, 2022
avatar
racofer: This is also affecting lgogdownloader, so it's not an issue with gogrepo. Something changed on GOG's end.
avatar
Geralt_of_Rivia: I just did some digging. The URLs to the CDN have changed.

So the URL manipulation to get the .xml with the file information instead of the actual file has to be rewritten. In my downloader it only took changing 2 lines of code so that everything was back to normal. So I guess gogrepo will also be updated pretty quickly.

However, some XML files also seem to be broken at the moment. For example the XML for /downloads/arma_gold_edition/en1installer3 is empty. It seems GOG also needs to do some fixing since their changes broke something.
Could you please post the two line changes here, so that we could each modify our local code until Kalanyr implements the fix?
Posted February 16, 2022
avatar
Geralt_of_Rivia: I just did some digging. The URLs to the CDN have changed.

So the URL manipulation to get the .xml with the file information instead of the actual file has to be rewritten. In my downloader it only took changing 2 lines of code so that everything was back to normal. So I guess gogrepo will also be updated pretty quickly.

However, some XML files also seem to be broken at the moment. For example the XML for /downloads/arma_gold_edition/en1installer3 is empty. It seems GOG also needs to do some fixing since their changes broke something.
avatar
mrkgnao: Could you please post the two line changes here, so that we could each modify our local code until Kalanyr implements the fix?
This please. Share the love.
Posted February 16, 2022
avatar
Geralt_of_Rivia: I just did some digging. The URLs to the CDN have changed.

So the URL manipulation to get the .xml with the file information instead of the actual file has to be rewritten. In my downloader it only took changing 2 lines of code so that everything was back to normal. So I guess gogrepo will also be updated pretty quickly.

However, some XML files also seem to be broken at the moment. For example the XML for /downloads/arma_gold_edition/en1installer3 is empty. It seems GOG also needs to do some fixing since their changes broke something.
avatar
mrkgnao: Could you please post the two line changes here, so that we could each modify our local code until Kalanyr implements the fix?
As I have said, I have written my own downloader. I'm not using gogrepo.

So I can't really tell you how to fix gogrepo because I don't know.
Posted February 16, 2022
avatar
mrkgnao: Could you please post the two line changes here, so that we could each modify our local code until Kalanyr implements the fix?
avatar
Geralt_of_Rivia: As I have said, I have written my own downloader. I'm not using gogrepo.

So I can't really tell you how to fix gogrepo because I don't know.
I misunderstood. I thought "my downloader" meant "my copy of gogrepo".

Still, since it's such a small change, can you post here the changes you made in your downloader (before and after of the two lines)? I (or someone else) may be able to apply it to gogrepo and share the fix here.
Posted February 16, 2022
high rated
avatar
Geralt_of_Rivia: As I have said, I have written my own downloader. I'm not using gogrepo.

So I can't really tell you how to fix gogrepo because I don't know.
avatar
mrkgnao: I misunderstood. I thought "my downloader" meant "my copy of gogrepo".

Still, since it's such a small change, can you post here the changes you made in your downloader (before and after of the two lines)? I (or someone else) may be able to apply it to gogrepo and share the fix here.
I think I've got it.

The issue is with how gogrepo grabs the .xml file with the md5 for each file in a game's manifest. If you check line 671, it generates an URL for the .xml file based on the URL of each file it downloads (.exe, .bin, etc). Something changed on GOG's end and now this no longer works.

I have managed to get the correct .xml file by changing the following at line 671:

tmp_md5_url = response.url.replace('?', '.xml?')

change it to:

tmp_md5_url = response.url + '.xml'

I have not yet tested for my entire library, but some minor testing with some random games seemed to work.

Edit: small correction, as I had some other changes on my script. The correct line is 671 and not 672 as I had initially mentioned.
Post edited February 16, 2022 by racofer
Posted February 16, 2022
avatar
racofer:
Thank you.
Posted February 16, 2022
avatar
Lebostein: Sorry, don't work:

18:02:12 | fatal...
Traceback (most recent call last):
File "gogrepoc.py", line 2777, in <module>
wakelock.take_wakelock()
File "gogrepoc.py", line 2685, in take_wakelock
self._PMerrcode, self._PMassertID = self._IOPMAssertionCreateWithName(a,self._kIOPMAssertionLevelOn,"gogrepoc")
File "gogrepoc.py", line 2643, in _IOPMAssertionCreateWithName
p_assert_name = self.raw_ptr(self._CFSTR(assert_name))
File "gogrepoc.py", line 2636, in _CFSTR
return CoreFoundation.CFStringCreateWithCString(None, py_string, CoreFoundation.kCFStringEncodingASCII)
ValueError: Expecting byte string of length 1, got a 'str'
avatar
Kalanyr: Ahh yes, there's a patch submitted for this on the repo (the Mac Wakelock code was only tested on Python 2), that I was hoping to put off until the next release. I guess I should do a hotfix for this then.
avatar
EdhelDil: Hello,

I did update, several times, latest one with:

E:\__games__\gog>python gogrepoc\gogrepoc.py update -full -ids
(which procedeed to update 1 by 1 every 606 games, but ended up with the same lack of setup*.exe as above, for the same games (cyberpunk 2077 and many others))

I also tried to "force" the update of cyberpunk 2077 and a few others:

for exemple:
E:\__games__\gog>python gogrepoc\gogrepoc.py update -full -ids anachronox # or other names
: no change, it doesn't find(?) or list the setup*exe (and .bin)

The manifest file is still not complete... And I have no clue as how to proceed further.

Edit: I moved away : the manifest file, the log, the manifest "resume" file, and tried again
It downloaded all infos from the games ( with options : update -full -ids ), and I end up with a file almost as big as before.
Then I downloaded with: download -lang en fr -os windows linux -skipgalaxy ici # ici is the subfolder download dir
: nothing new came up... still missing several setup*.exe
avatar
Kalanyr: This sounds like your manifest doesn't have the installer files at all.

gogrepoc.py update -full -os windows linux -lang en fr

Should effectively build a new manifest with everything in it. IIRC the default behaviour for update if -os and -lang isn't specified is to try for your OS (but the detection seems kinda dodgy on anything by Windows) + your system language, defaulting to English. And new updates can clobber old ones.

(it's possible fr is the wrong abbreviation for french , since gogrepo uses the GOG codes which aren't standard but I think fr is correct).

Download should try and grab everything in the manifest by default IIRC (sorry, I'm away so don't have my files to check) but you can force it with:

gogrepoc.py download "D:\TargetDir" -os windows linux -lang en fr

Should download to whatever path you specify in "D:\TargetDir"

If you're comfortable doing so I'd open up the manifest file and see if it actually has any executable files listed. It's possible the handful you are seeing are actually the handful of unsupported special installers that GOG provides as extras.
thx a lot for your reply.

I tried: to delete the manifest (and manifest-resume) files and try with your suggestion

it seems to choke on a lot of xml files? am I missing something ?

in a windows CMD.exe window:

E:\__games__\gog>del gog-manifest.dat

E:\__games__\gog>del gog-resume-manifest.dat

E:\__games__\gog>del gogrepo.log

E:\__games__\gog>python gogrepoc\gogrepoc.py update -full -os windows linux -lang en fr
03:20:41 | loading local manifest...
03:20:41 | loading local resume manifest...
03:20:41 | fetching game product data (page 1)...
03:20:41 | fetching game product data (page 2 / 7)...
03:20:42 | fetching game product data (page 3 / 7)...
03:20:42 | fetching game product data (page 4 / 7)...
03:20:43 | fetching game product data (page 5 / 7)...
03:20:43 | fetching game product data (page 6 / 7)...
03:20:43 | fetching game product data (page 7 / 7)...
03:20:44 | found 610 games !!!!!!!!
03:20:44 | saving resume manifest...
03:20:44 | saved resume manifest
03:20:44 | ( 1 / 610) fetching game details for 8th_wonder_of_the_world...
03:20:54 | xml parsing error occurred trying to get md5 data for setup_8th_wonder_2.1.0.9.exe
03:20:55 | ( 2 / 610) fetching game details for a_new_beginning...
03:20:57 | xml parsing error occurred trying to get md5 data for setup_a_new_beginning_final_cut_2.0.4.0395_(27805).exe
03:23:18 |
Traceback (most recent call last):
File "E:\__games__\gog\gogrepoc\gogrepoc.py", line 716, in filter_downloads
fetch_file_info(d, True,updateSession)
File "E:\__games__\gog\gogrepoc\gogrepoc.py", line 673, in fetch_file_info
shelf_etree = xml.etree.ElementTree.fromstring(md5_response.content)
File "C:\Program Files\Python310\lib\xml\etree\ElementTree.py", line 1347, in XML
parser.feed(text)
OverflowError: size does not fit in an int
03:23:18 | failed to fetch https://www.gog.com/downloads/a_new_beginning/en1installer1 because of non-HTTP Error
03:23:23 | xml parsing error occurred trying to get md5 data for patch_a_new_beginning_-_final_cut_1.0_(19838)_to_2.0.4.0395_(27805).exe
03:23:25 | ( 3 / 610) fetching game details for afterlife...
03:23:32 | xml parsing error occurred trying to get md5 data for setup_afterlife_2.0.0.4.exe
03:23:50 | xml parsing error occurred trying to get md5 data for gog_afterlife_2.2.0.8.sh
03:24:17 | xml parsing error occurred trying to get md5 data for setup_afterlife_french_2.0.0.4.exe
03:24:45 | xml parsing error occurred trying to get md5 data for gog_afterlife_french_2.2.0.8.sh
03:24:45 | ( 4 / 610) fetching game details for age_of_wonders_3...
03:24:48 | xml parsing error occurred trying to get md5 data for setup_age_of_wonders_iii_1.802_fix_(28713).exe
03:31:02 | exiting...
Exception ignored in atexit callback: <function shutdown at 0x0000015649E58AF0>
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\logging\__init__.py", line 2174, in shutdown
h.close()
File "C:\Program Files\Python310\lib\logging\__init__.py", line 1183, in close
stream.close()
KeyboardInterrupt:

(I didn't have those xml parsing errors with my previous tries, but those were with a different command that, as you pointed out, probably didn't try to fetch the binaries themselves)
Post edited February 16, 2022 by EdhelDil
Posted February 16, 2022
avatar
EdhelDil:
You can find a solution for the xml parsing issue two posts above yours, here:
https://www.gog.com/forum/general/gogrepopy_python_script_for_regularly_backing_up_your_purchased_gog_collection_for_full_offline_e/post2876
Posted February 16, 2022
avatar
mrkgnao: I misunderstood. I thought "my downloader" meant "my copy of gogrepo".

Still, since it's such a small change, can you post here the changes you made in your downloader (before and after of the two lines)? I (or someone else) may be able to apply it to gogrepo and share the fix here.
avatar
racofer: I think I've got it.

The issue is with how gogrepo grabs the .xml file with the md5 for each file in a game's manifest. If you check line 671, it generates an URL for the .xml file based on the URL of each file it downloads (.exe, .bin, etc). Something changed on GOG's end and now this no longer works.

I have managed to get the correct .xml file by changing the following at line 671:

tmp_md5_url = response.url.replace('?', '.xml?')

change it to:

tmp_md5_url = response.url + '.xml'

I have not yet tested for my entire library, but some minor testing with some random games seemed to work.

Edit: small correction, as I had some other changes on my script. The correct line is 671 and not 672 as I had initially mentioned.
OK I can verify this fix....

My game library manifest was up-to-date and everything verified as of 8 or 9AM today 2/15/22. Late this afternoon I noticed a game had been flagged as updated and when I updated the manifest I was getting a bunch of errors for some of the files:

xml parsing error occurred trying to get md5 data for <insert filename here>

I didn't think too much about it and later this evening I saw Cyberpunk 2077 had the 1.5 patch released and the game wasn't flagged as being updated (yet). So I tried to update the manifest and it was spewing more xml parsing errors trying to get md5 data for the game....only it hung up and started eating RAM on my system. It eventually crashed HARD after consuming more than 80% of my system's memory.

I tried a couple of times to update and kept having the same errors and RAM eating problem. I saw it kept seeing an incomplete update detected and resuming. Thinking maybe that was partially the issue I deleted gog-resume-manifest.dat and tried again only to have no joy.

Then I screwed up. I went to delete gog-resume-manifest.dat again and deleted gog-manifest.dat. Yup, I killed my manifest. Side note: linux command line and not Windows so no recycle bin...so, basically, I was screwed.

I restored a fairly "recent" backup of my manifest from 2/4/22 and tried to update it thinking it'll just pull down the info for any new games and I can manually specify any updated games (18 new games and 7 games had updated, some multiple times, since that backup manifest was made). One of those updated games is the one I noticed throwing xml errors earlier this afternoon. But, I figured if I could update the manifest and it see the 18 new and if I specified 6 of the 7 manually and I run a verify against it then everything would be back to as it was this morning before any issues occurred.

Only...MORE xml parsing errors on stuff I knew was fine!! So I checked this thread and found the above fix. I made the change and after one run it picked up the 18 new games, no downloads needed, and verification of those files passed (again). Then I manually specified the 6 titles and the manifest updated, no downloads needed, and verification passed yet again for those files. And, yes, I realize if some games updated multiple times that it is only verifying against the latest version but since I verify after every download I wasn't concerned. I was more interested in an up-to-date manifest with files verified for it.

This leads me to my last known updated game: Northgard, which is the one that set about this whole issue. And the following occurs:

22:38:12 | (1 / 1) fetching game details for northgard...
22:38:44 | xml parsing error occurred trying to get md5 data for setup_northgard_-_ratatoskr_clan_of_the_squirrel_2.7.5.24893_(64bit)_(53701).exe
22:39:35 | request failed: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out. (3 retries left) -- will retry in 5s...
22:40:15 | request failed: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out. (2 retries left) -- will retry in 5s...
22:40:35 | xml parsing error occurred trying to get md5 data for setup_northgard_-_himminbrjotir_clan_of_the_ox_2.7.5.24893_(64bit)_(53701).exe
22:40:50 | -> changelog was updated

So, while I was able to verify the fix (at least partially) and get back to a known state from this morning, clearly there are still some problems. Currently, Cyberpunk 2077 updated the manifest just fine but is throwing more xml parsing errors when trying to download:

23:26:46 | ------------------------------------------------------------
23:26:46 | attempting preallocating '2222944' bytes for './!downloading/cyberpunk_2077_game/setup_cyberpunk_2077_build_3836782change_4719897_dllfix_(64bit)_(53680).ex e' using posix_fallocate
23:26:46 | attempting preallocating '4292751358' bytes for './!downloading/cyberpunk_2077_game/setup_cyberpunk_2077_build_3836782change_4719897_dllfix_(64bit)_(53680)-1. bin' using posix_fallocate
23:26:47 | xml parsing error occurred trying to get md5 data for https://gog-cdn-lumen-cp77.secure2.footprint.net/&lt;snip&gt;/setup_cyberpunk_2077_build_3836782change_4719897_dllfix_%2864bit%29_%2853680%29.exe[/url]

Not holding my breath on this one...

edit: Yep, the script crashed hard spewing pages of errors. Looks like a MemoryError occurred. I'll have to reattempt this tomorrow on another system using Windows.
Post edited February 16, 2022 by kmyst
Posted February 16, 2022
avatar
kmyst: So, while I was able to verify the fix (at least partially) and get back to a known state from this morning, clearly there are still some problems. Currently, Cyberpunk 2077 updated the manifest just fine but is throwing more xml parsing errors when trying to download:
The xml parsing issue during download can be fixed by replacing line 641:
chunk_url = response.url.replace('?', '.xml?')
with:
chunk_url = response.url + '.xml'
Pages:
1
50
100
150
190
191
192
193
194
250
272
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4078 posts)(4078 posts)
(4078 posts)