It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Just popping in here to say this tool is wonderful. Recently used it to make a more consistent (and scalable, relatively speaking... it is Python after all) backup of all my shit. 1.5 TiB download was very smooth.
Someone from GOG mentioned that gogrepoc is using the wrong endpoint for that, he said that it should be downloadlink instead of downloads.

I did try to debug it yesterday, but couldn't locate the exactly place in the script.
avatar
coffeecup: Someone from GOG mentioned that gogrepoc is using the wrong endpoint for that, he said that it should be downloadlink instead of downloads.

I did try to debug it yesterday, but couldn't locate the exactly place in the script.
It tries both, though I can swap the order if that's prefered ( downlink is currently the fallback because download used to be the only one).

I'll run some tests tomorrow and see if I can work out what's going on
avatar
coffeecup: Well, I did the nuclear option; removed the old installation, did a fresh git clone, a new login and then update:

I only censored the hash values of the DL urls and personal information.

It seems that there is a problem in gogrepoc, because I dont' t have a problem with lgogdownloader and gogg (the new in town).

$ ~/bin/gog> git clone https://github.com/Kalanyr/gogrepoc
Cloning into 'gogrepoc'...
remote: Enumerating objects: 504, done.
remote: Counting objects: 100% (135/135), done.
remote: Compressing objects: 100% (19/19), done.
remote: Total 504 (delta 119), reused 116 (delta 116), pack-reused 369 (from 1)
Receiving objects: 100% (504/504), 247.29 KiB | 3.34 MiB/s, done.
Resolving deltas: 100% (202/202), done.
$ ~/bin/gog> cd gogrepoc/
$ ~/bin/gog/gogrepoc> mcedit -d gogrepoc.py

$ ~/bin/gog/gogrepoc> ./gogrepoc.py login
Username: very.secret.shopping@example.com
Password:
16:26:44 | attempting Galaxy login as 'very.secret.shopping@example.com' ...
16:26:45 | saving token...
16:26:45 | saved token
16:26:45 | exiting...
$ ~/bin/gog/gogrepoc> ./gogrepoc.py update
16:26:53 | loading local manifest...
16:26:53 | loading token...
16:26:53 | loading local resume manifest...
16:26:53 | fetching game product data (page 1)...
16:26:55 | fetching game product data (page 2 / 35)...
...
16:27:39 | fetching game product data (page 35 / 35)...
16:27:40 | found 3482 games !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
16:27:40 | saving resume manifest...
16:27:41 | saved resume manifest
16:27:41 | ( 1 / 3482) fetching game details for 140_game...
16:27:42 | ( 2 / 3482) fetching game details for 1428_shadows_over_silesia...
16:27:43 | failed to fetch https://www.gog.com/downloads/1428_shadows_over_silesia_bonus_stories/94168 because of non-HTTP Error
16:27:43 | The handled exception was:
16:27:43 |
Traceback (most recent call last):
File "~/bin/gog/gogrepoc/./gogrepoc.py", line 1244, in filter_extras
fetch_file_info(d, False,save_md5_xml,updateSession)
File "~/bin/gog/gogrepoc/./gogrepoc.py", line 1088, in fetch_file_info
d.raw_updated = d.gog_data.headers["Last-Modified"]
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'Last-Modified'
16:27:43 | End exception report.
16:27:43 | request failed: 403 Client Error: Forbidden for url: https://cdn.gog.com/secure/1428_shadows_over_silesia_bonus_stories/1428_shadows_over_silesia_bonus_stories.zip?1223fb43868c5f174d699acf5f3ffcf657055d06f0acd6c5e2cf9203201871e3c009c4b2aa9173ddd8e5bfd0f12768f5f6410e6e2811d2de7088c325fe8c2a6488fc1f6345ff87890dc0a447692b728051784a0c4892fb7e31e20c2b3c2645179399eecbc77076ca1c38c2dacc9498c05c2f6374404ac3062f9f0061369bb75d489cf8751720f39f2947f56153b0cc30a7e591ea34cc998e07&fileExtForIe=.exe. will not retry.
16:27:43 | failed to fetch https://www.gog.com/downlink/1428_shadows_over_silesia_bonus_stories/94168
16:27:43 | Could not fetch file info so using canonical link: https://www.gog.com/downloads/1428_shadows_over_silesia_bonus_stories/94168
16:27:44 | failed to fetch https://www.gog.com/downloads/1428_shadows_over_silesia_soundtrack/94120 because of non-HTTP Error
16:27:44 | The handled exception was:
16:27:44 |
Traceback (most recent call last):
File "~/bin/gog/gogrepoc/./gogrepoc.py", line 1244, in filter_extras
fetch_file_info(d, False,save_md5_xml,updateSession)
File "~/bin/gog/gogrepoc/./gogrepoc.py", line 1088, in fetch_file_info
d.raw_updated = d.gog_data.headers["Last-Modified"]
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'Last-Modified'
16:27:44 | End exception report.
16:27:44 | request failed: 403 Client Error: Forbidden for url: https://cdn.gog.com/secure/1428_shadows_over_silesia_soundtrack/1428_shadows_over_silesia_soundtrack_mp3.zip?1223fb43868c5f174d699acf593ffcf6a5491c40e4cb456a2494ad9082d38fb4b30df34b6371a64d1db24bf7af9b723acc252ed4597ad77ce55a55c5858c7c784edf98e4d29b10ec91b6a639f8a5a1b27cebc69106cb8fd9fa141d5848341f2e2c50541c359f8974f3b49c4188bd86b79887e4c73a086530c866b4bab75c4c07f2b1c8a0859fbdf920a92600c6b5c15ce8b4aef07fc&fileExtForIe=.exe. will not retry.
16:27:44 | failed to fetch https://www.gog.com/downlink/1428_shadows_over_silesia_soundtrack/94120
16:27:44 | Could not fetch file info so using canonical link: https://www.gog.com/downloads/1428_shadows_over_silesia_soundtrack/94120
16:27:44 | ( 3 / 3482) fetching game details for 15_days...
16:27:45 | failed to fetch https://www.gog.com/downloads/15_days/79876 because of non-HTTP Error
16:27:45 | The handled exception was:
16:27:45 |
Traceback (most recent call last):
File "~/bin/gog/gogrepoc/./gogrepoc.py", line 1244, in filter_extras
fetch_file_info(d, False,save_md5_xml,updateSession)
File "~/bin/gog/gogrepoc/./gogrepoc.py", line 1088, in fetch_file_info
d.raw_updated = d.gog_data.headers["Last-Modified"]
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'Last-Modified'
16:27:45 | End exception report.
^C16:27:45 | exiting...
Okay, did some testing and I can't duplicate this. The problem seems to be that gog isn't returning a Last-Modified header for you but it is in my tests. The other issue is that GOG is consistently 403ing you afterwards , which suggests you're hitting some kind of automated No No switch for doing something unacceptable, but I don't think you're going to be anymore aggressive than my Full Update from a few days ago was.

Are you running Linux or Mac and if Mac is it an X86 Mac or one of the newer ARM ones ? I think I might need to test on the proper OS to get a better idea of what's happening.

ETA - Okay, speak of the devil, I did just get this error for parkitect wallpaper and only that. Going to go poke and see what's going on.
ETA2 - Problem identified, some of the servers are returning headers with last-modified instead of Last-Modified, this is non-standard at the least and I think in violation of the Header standards but GOG technically violates those by returning different information for HEAD vs GET on the same URI anyway I guess.
Post edited January 19, 2025 by Kalanyr
Okay, fixed the issue I think. Going to go check and see how far my local copy has drifted from the uploaded one and decide if I should emergency hotfix or throw it here for some testing first.

ETA - Uploaded to dev. If some people would like to give it a work out that would be great. Given the potential severity of the issue I'll push it to master in 3 days if I don't get any feedback and sooner if I get some feedback indicating there's no problems.
Post edited January 19, 2025 by Kalanyr
avatar
Kalanyr: ETA2 - Problem identified, some of the servers are returning headers with last-modified instead of Last-Modified, this is non-standard at the least and I think in violation of the Header standards but GOG technically violates those by returning different information for HEAD vs GET on the same URI anyway I guess.
Seriously??? Can you provide an example URL where GOG delivers different data for HEAD than for GET?
avatar
Kalanyr: ETA2 - Problem identified, some of the servers are returning headers with last-modified instead of Last-Modified, this is non-standard at the least and I think in violation of the Header standards
Seems not, from RFC 2616:

https://www.rfc-editor.org/rfc/rfc2616#section-4.2

4.2 Message Headers

HTTP header fields, which include general-header (section 4.5),
request-header (section 5.3), response-header (section 6.2), and
entity-header (section 7.1) fields, follow the same generic format as
that given in Section 3.1 of RFC 822 [9]. Each header field consists
of a name followed by a colon (":") and the field value. Field names
are case-insensitive
avatar
Kalanyr: ETA2 - Problem identified, some of the servers are returning headers with last-modified instead of Last-Modified, this is non-standard at the least and I think in violation of the Header standards but GOG technically violates those by returning different information for HEAD vs GET on the same URI anyway I guess.
avatar
Geralt_of_Rivia: Seriously??? Can you provide an example URL where GOG delivers different data for HEAD than for GET?
All file downloads ( for existing files ) return different Header data from Head than from Get ( even with a body size of 0 requested ) from the generic download link for that file ( I assume this is because there's some kind of redirection happening with GET that doesn't happen with HEAD so HEAD returns the data for the generic link rather than the file URI ).
Post edited January 19, 2025 by Kalanyr
avatar
Geralt_of_Rivia: Seriously??? Can you provide an example URL where GOG delivers different data for HEAD than for GET?
avatar
Kalanyr: All file downloads ( for existing files ) return different Header data from Head than from Get ( even with a body size of 0 requested ) from the generic download link for that file ( I assume this is because there's some kind of redirection happening with GET that doesn't happen with HEAD so HEAD returns the data for the generic link rather than the file URI ).
Have you configured cURL (or whatever library you use) to automatically follow redirections? If you did so for your GET request but not for your HEAD request than you might get different results. But if you made sure that the setting is identical that you should get the same result.

I just checked with 5 files and always got the same result.
avatar
Kalanyr: All file downloads ( for existing files ) return different Header data from Head than from Get ( even with a body size of 0 requested ) from the generic download link for that file ( I assume this is because there's some kind of redirection happening with GET that doesn't happen with HEAD so HEAD returns the data for the generic link rather than the file URI ).
avatar
Geralt_of_Rivia: Have you configured cURL (or whatever library you use) to automatically follow redirections? If you did so for your GET request but not for your HEAD request than you might get different results. But if you made sure that the setting is identical that you should get the same result.

I just checked with 5 files and always got the same result.
Tested with the python requests library with the same adjustments to settings for each (both called from the same session) , seems like there's some kind of different default for each then. One more thing to look into but it'll simplify a chunk of the code if I can get work out why that is.
avatar
Kalanyr: Are you running Linux or Mac and if Mac is it an X86 Mac or one of the newer ARM ones ? I think I might need to test on the proper OS to get a better idea of what's happening.
With the dev branch, everything run through, but it took about 80 minutes for a full scan. Didn't see anything related to this problem any more. There were some errors, but that were files that should have been here, but aren't for some reasons only the publishers know, mostly related to linux builds of games that have fallen wayside.

Oh, and I use Arch btw;

just kidding,

$ cat /etc/os-release
PRETTY_NAME="openSUSE Tumbleweed"
VERSION_ID="20250118"
HOME_URL="https://www.opensuse.org"
DOCUMENTATION_URL="[url=https://en.opensuse.org/Portal:Tumbleweed]https://en.opensuse.org/Portal:Tumbleweed[/url]"

$ uname -a
Linux riser.local.schatzl.me 6.12.9-1-default #1 SMP PREEMPT_DYNAMIC Fri Jan 10 08:30:10 UTC 2025 (0ae2136) x86_64 x86_64 x86_64 GNU/Linux

$ python3
Python 3.11.11 (main, Dec 04 2024, 21:44:34) [GCC] on linux

I linked a pastebin as a discord DM for perusal.
Post edited January 20, 2025 by coffeecup
I also noticed that if I do

$ gogrepoc update
$ gogrepoc download

does NOT include any installer in the metadata, but I have to explicitely set

$ gogrepoc update -os something

that it even tries to grab any installer or galaxy games....

Can you change that, if I just run "gogrepoc update" that it reads in any available installers it can and not only the extras?

Sorting out the os, language etc, should be done at the download stage.

Oh, can you change the metadata language default to english and not the geo-fenced version, so the genres etc are made sure that they are (default) in english?

Thanks ;D
avatar
coffeecup: I also noticed that if I do

$ gogrepoc update
$ gogrepoc download

does NOT include any installer in the metadata, but I have to explicitely set

$ gogrepoc update -os something

that it even tries to grab any installer or galaxy games....

Can you change that, if I just run "gogrepoc update" that it reads in any available installers it can and not only the extras?

Sorting out the os, language etc, should be done at the download stage.

Oh, can you change the metadata language default to english and not the geo-fenced version, so the genres etc are made sure that they are (default) in english?

Thanks ;D
The defaults are to use the OS you're running on ( so should be LInux for you ) and the system language.

I should probably add a config file to let peope set their own defaults at some point.
avatar
coffeecup: Traceback (most recent call last):
File "/home/user/bin/gog/gogrepoc/./gogrepoc.py", line 1244, in filter_extras
fetch_file_info(d, False,save_md5_xml,updateSession)
File "/home/user/bin/gog/gogrepoc/./gogrepoc.py", line 1088, in fetch_file_info
d.raw_updated = d.gog_data.headers["Last-Modified"]
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'Last-Modified'
19:15:32 | End exception report.
I am now experiencing the same exact thing for random games, but slightly different lines.

Traceback (most recent call last):
File "C:\gogrepoc-master\gogrepoc.py", line 1243, in filter_extras
fetch_file_info(d, False,save_md5_xml,updateSession)
File "C:\gogrepoc-master\gogrepoc.py", line 1087, in fetch_file_info
d.raw_updated = d.gog_data.headers["Last-Modified"]
KeyError: 'Last-Modified'
01:39:31 | End exception report.

Did GOG change something? What the heck is going on?
Post edited January 22, 2025 by SargonAelther
avatar
SargonAelther: Did GOG change something? What the heck is going on?
Something really seems to be going on. I'm suddenly getting 404 for most of the .xml files.