It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
avatar
mrkgnao: Correction:
Files are downloaded to !downloading, moved (one file at a time) to !provisional whenever a single file download has been completed, and then moved (in bulk) to their final resting place when the entire operation is done.

When the entire download operation is over, !provisional and !downloading should be empty. Is that not the case?
avatar
timppu: I wasn't aware of this either, unless it is a very recent change. I thought incomplete files are in the !downloading and are moved to the main download directory when they are complete (each file one by one).

Any idea why there is now the extra step and directory, !provisional?
Tracking potential updates by Timestamp requires me to note a files new timestamp state when it's downloaded, writing to the manifest requires a global read / write lock on the manifest and that was happening too often with downloads and slowing things down too much, so !provisional lets me mark things as "done" in a resume friendly way without touching the manifest and the manifest gets bulk updated at the end ( when everything is moved to its final destination) requiring only a single lock.

avatar
mrkgnao: It's fairly new. You will have to ask Kalanyr for the reason.

I expect you should be ok then. How do you stop a download in the middle?
avatar
paladin181: I close the cmd window.
I recommend whatever your OS's interrupt ( eg Ctrl-C on Windows ) is but IIRC that's the default behaviour of closing the window on Windows. There's an interrupt handler that blocks the interrupt if you're doing something incredibly stupid like trying to quit when the manifest is only partially written.
Post edited December 30, 2024 by Kalanyr
avatar
Kalanyr: Tracking potential updates by Timestamp requires me to note a files new timestamp state when it's downloaded, writing to the manifest requires a global read / write lock on the manifest and that was happening too often with downloads and slowing things down too much, so !provisional lets me mark things as "done" in a resume friendly way without touching the manifest and the manifest gets bulk updated at the end ( when everything is moved to its final destination) requiring only a single lock.

I recommend whatever your OS's interrupt ( eg Ctrl-C on Windows ) is but IIRC that's the default behaviour of closing the window on Windows. There's an interrupt handler that blocks the interrupt if you're doing something incredibly stupid like trying to quit when the manifest is only partially written.
As for provisional, that makes sense. As long as items are in provisional, the download script will see them there on its initial scan, and move them when the whole operation properly completes whether they were added in this session or a previous one, yes?

I'm not editing the Manifest. I don't interrupt that. That only took a few hours to build, and honestly, that file hasn't edited since I built it (26.12), according to "last modified" in properties. I wouldn't interrupt the mainfest edit, but if I did, I'd wait a day or two and re-run a clean manifest update.
avatar
Kalanyr: Tracking potential updates by Timestamp requires me to note a files new timestamp state when it's downloaded, writing to the manifest requires a global read / write lock on the manifest and that was happening too often with downloads and slowing things down too much, so !provisional lets me mark things as "done" in a resume friendly way without touching the manifest and the manifest gets bulk updated at the end ( when everything is moved to its final destination) requiring only a single lock.

I recommend whatever your OS's interrupt ( eg Ctrl-C on Windows ) is but IIRC that's the default behaviour of closing the window on Windows. There's an interrupt handler that blocks the interrupt if you're doing something incredibly stupid like trying to quit when the manifest is only partially written.
avatar
paladin181: As for provisional, that makes sense. As long as items are in provisional, the download script will see them there on its initial scan, and move them when the whole operation properly completes whether they were added in this session or a previous one, yes?

I'm not editing the Manifest. I don't interrupt that. That only took a few hours to build, and honestly, that file hasn't edited since I built it (26.12), according to "last modified" in properties. I wouldn't interrupt the mainfest edit, but if I did, I'd wait a day or two and re-run a clean manifest update.
Yes, correct, it moves anything from the current download set ( and if you're using the automatic resume you can't lose anything that isn't somehow broken that way ).

I don't mean you personally I mean, it's possible to try and quit gogrepc while *it's* writing the manifest out which is extremely bad ( I assume it isn't updated because you haven't actually finished a full set yet ), so I block quitting in that circumstance as long as an interrupt is being used to ask to quit politely..
avatar
Kalanyr: Yes, correct, it moves anything from the current download set ( and if you're using the automatic resume you can't lose anything that isn't somehow broken that way ).

I don't mean you personally I mean, it's possible to try and quit gogrepc while *it's* writing the manifest out which is extremely bad ( I assume it isn't updated because you haven't actually finished a full set yet ), so I block quitting in that circumstance as long as an interrupt is being used to ask to quit politely..
Yes, I realize that you meant gogrepoc editing it. I just meant that none of the operations I quit during are performing updates to the manifest at that time. I understand all that. Thanks for the concern. I am outputting to a log file that I actively check at the time before quitting out, but I will use Ctrl-C as a double check method. Thanks for the info.
avatar
Kalanyr: Tracking potential updates by Timestamp requires me to note a files new timestamp state when it's downloaded, writing to the manifest requires a global read / write lock on the manifest and that was happening too often with downloads and slowing things down too much, so !provisional lets me mark things as "done" in a resume friendly way without touching the manifest and the manifest gets bulk updated at the end ( when everything is moved to its final destination) requiring only a single lock.
Sounds reasonable then.

I've still been trying to disable the preallocation on the script since it still doesn't work on Linux if the target filesystem is NTFS (an USB hard drive), but it seems preallocation stuff is now in quite many places in the script, at least if you just search for the term within the script... Gah!

I tried to add an extra condition that "if preallocation == 1 then do the following preallocation stuff, otherwise not" (and I define in the beginning of the script whether the value of preallocation is 1 or something else...) but apparently it would need to be added to quite many places.

It would be nice if preallocation was optional since it seems to cause problems in certain cases (someone else also reported problems with it when trying to download on his NAS system).
Post edited December 30, 2024 by timppu
Hi Folks,,

Updating my machine and just wanted to check...

It looks like the last update to the script on github was Aug. 11th. That's the latest version, right?

Thanks
avatar
UrsaCorvalis: Hi Folks,,

Updating my machine and just wanted to check...

It looks like the last update to the script on github was Aug. 11th. That's the latest version, right?

Thanks
Assuming you are getting the Master branch, the last update was 14 July 2024.

The 11 August 2024 update was done to the Dev branch and not yet merged into the Master branch.
Post edited January 05, 2025 by mrkgnao
avatar
mrkgnao: Assuming you are getting the Master branch, the last update was 14 July 2024.

The 11 August 2024 update was done to the Dev branch and not yet merged into the Master branch.
didn't catch that. Thanks
Next update will probably be early February, probably around the 7th.

I need to patch that thing causing the crash if you're !md5_xmls is in an unexpected location, clean up the logic on the Timestamp update checking for Extras (turns out the GOG servers have inconsistent dates for some of them, so I'm going to have to check it to check for a *newer* date rather than a *different* date, not check timestamps for fake entries (unreleased / unavailable) and ignore None -> None where possible ), and make pre-allocation optional (if I get the chance I'll improve the auto-detection too, so that Linux systems don't try to pre-allocation NFTS drives or Linux FS that don't have useful pre-allocation ( spare preallocation is pointless for this purpose ) but no promises on that one, detecting underlying FS is a pain in Python )
avatar
Kalanyr: and make pre-allocation optional (if I get the chance I'll improve the auto-detection too, so that Linux systems don't try to pre-allocation NFTS drives or Linux FS that don't have useful pre-allocation ( spare preallocation is pointless for this purpose ) but no promises on that one, detecting underlying FS is a pain in Python )
I guess they are exceptional cases where preallocation does not work, and I am unsure if we have even identified all the possible cases where it might fail.

So IMHO it being just selectable (ie. can be easily disabled within the script, or an extra option to not use it) would be enough, and it could be used in any future case if preallocation becomes a problem.

To me it appears one problem is indeed if Linux is trying to write to a NTFS partition (then the script gets stuck to the preallocation; works fine without it), but I recall someone else reported a similar problem when pointing gogrepo to some NAS drive (I don't remember if that NAS drive was NTFS and if the source OS was Linux there too...).
Post edited January 08, 2025 by timppu
avatar
Kalanyr: Yes, correct, it moves anything from the current download set ( and if you're using the automatic resume you can't lose anything that isn't somehow broken that way ).

I don't mean you personally I mean, it's possible to try and quit gogrepc while *it's* writing the manifest out which is extremely bad ( I assume it isn't updated because you haven't actually finished a full set yet ), so I block quitting in that circumstance as long as an interrupt is being used to ask to quit politely..
avatar
paladin181: Yes, I realize that you meant gogrepoc editing it. I just meant that none of the operations I quit during are performing updates to the manifest at that time. I understand all that. Thanks for the concern. I am outputting to a log file that I actively check at the time before quitting out, but I will use Ctrl-C as a double check method. Thanks for the info.
On this topic, I can confirm that closing the window on Windows does not in fact send an interrupt request, in maybe related news it's time for me to restore my manifest from my last backup and do a full update *sigh*. I guess I should put write manifest/resume_manifest to a temporary file and swap it if it only if it completes successfully to the ToDo list.
Post edited January 14, 2025 by Kalanyr
avatar
Kalanyr: On this topic, I can confirm that closing the window on Windows does not in fact send an interrupt request, in maybe related news it's time for me to restore my manifest from my last backup and do a full update *sigh*. I guess I should put write manifest/resume_manifest to a temporary file and swap it if it only if it completes successfully to the ToDo list.
Yep. You can never underestimate the power of dumb people (like me) to break things!!
I wanted to try out gogrepoc and did get the most recent commit https://github.com/Kalanyr/gogrepoc/commit/c453e922f94b3b55e67b65ad583320ff5fd08042 and Python 3.11, for trying out some stuff.

./gogrepoc.py update
19:15:17 | loading local manifest...
19:15:17 | loading token...
19:15:17 | loading local resume manifest...
19:15:17 | incomplete update detected, resuming...
19:15:17 | saving resume manifest...
19:15:18 | saved resume manifest
19:15:18 | ( 1 / 3431) fetching game details for aer...
19:15:20 | ( 2 / 3431) fetching game details for afterlife...
19:15:22 | ( 3 / 3431) fetching game details for against_the_moon_prologue...
19:15:23 | ( 4 / 3431) fetching game details for agarest_generations_of_war...
19:15:30 | failed to fetch https://www.gog.com/downloads/agarest_generations_of_war_collectors_edition_upgrade/38083 because of non-HTTP Error
19:15:30 | The handled exception was:
19:15:30 |
Traceback (most recent call last):
File "/home/user/bin/gog/gogrepoc/./gogrepoc.py", line 1244, in filter_extras
fetch_file_info(d, False,save_md5_xml,updateSession)
File "/home/user/bin/gog/gogrepoc/./gogrepoc.py", line 1088, in fetch_file_info
d.raw_updated = d.gog_data.headers["Last-Modified"]
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'Last-Modified'
19:15:30 | End exception report.
19:15:31 | request failed: 403 Client Error: Forbidden for url: https://cdn.gog.com/secure/agarest/extras/collectors_content/agarest_ce_digital_artbook.zip?1223fb43868c5f174d699acd5e3ffcf6201d8cfdec2a089be79108b742fa1f3738dd9e92b9f7fdf60dbd9ac9cb6342dbabe5da6aeb2bd0ca4c4416bdc0a687cbd93e9bfd5b9d681a4a1dfae7c1277154090f0975d8fd437ec94c881d9492946a899f19326de686947613e5357af3f2cba601b217c05c87e90ac8ae4f508312ce5c21542d1cf16fb&fileExtForIe=.exe. will not retry.
19:15:31 | failed to fetch https://www.gog.com/downlink/agarest_generations_of_war_collectors_edition_upgrade/38083
19:15:31 | Could not fetch file info so using canonical link: https://www.gog.com/downloads/agarest_generations_of_war_collectors_edition_upgrade/38083
19:15:31 | ( 5 / 3431) fetching game details for agarest_generations_of_war_2...
19:15:32 | failed to fetch https://www.gog.com/downloads/agarest_generations_of_war_2/58103 because of non-HTTP Error
19:15:32 | The handled exception was:
19:15:32 |
Traceback (most recent call last):
File "/home/user/bin/gog/gogrepoc/./gogrepoc.py", line 1244, in filter_extras
fetch_file_info(d, False,save_md5_xml,updateSession)
File "/home/user/bin/gog/gogrepoc/./gogrepoc.py", line 1088, in fetch_file_info
d.raw_updated = d.gog_data.headers["Last-Modified"]
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'Last-Modified'
19:15:32 | End exception report.
^C19:15:32 | exiting...

Is this reproducible, that the script throws an exception on trying to download metadata for my games, I think this is only if it tries to retrieve the data for the extras (manuals, wallpapers etc), not the installer itself.

It was mentioned that the relevant endpoint "downloads" was changed to "downlink", but I couldn't fully put my finger on it during debugging the script, could trace it back in the function "def filter_extras()" line 1195ff, but not sure how to solve it.

Can anyone confirm that, that the metadata loading of extras throws this exception?
Post edited January 17, 2025 by coffeecup
avatar
coffeecup: I wanted to try out gogrepoc and did get the most recent commit https://github.com/Kalanyr/gogrepoc/commit/c453e922f94b3b55e67b65ad583320ff5fd08042 and Python 3.11, for trying out some stuff.

./gogrepoc.py update
19:15:17 | loading local manifest...
19:15:17 | loading token...
19:15:17 | loading local resume manifest...
19:15:17 | incomplete update detected, resuming...
19:15:17 | saving resume manifest...
19:15:18 | saved resume manifest
19:15:18 | ( 1 / 3431) fetching game details for aer...
19:15:20 | ( 2 / 3431) fetching game details for afterlife...
19:15:22 | ( 3 / 3431) fetching game details for against_the_moon_prologue...
19:15:23 | ( 4 / 3431) fetching game details for agarest_generations_of_war...
19:15:30 | failed to fetch https://www.gog.com/downloads/agarest_generations_of_war_collectors_edition_upgrade/38083 because of non-HTTP Error
19:15:30 | The handled exception was:
19:15:30 |
Traceback (most recent call last):
File "/home/user/bin/gog/gogrepoc/./gogrepoc.py", line 1244, in filter_extras
fetch_file_info(d, False,save_md5_xml,updateSession)
File "/home/user/bin/gog/gogrepoc/./gogrepoc.py", line 1088, in fetch_file_info
d.raw_updated = d.gog_data.headers["Last-Modified"]
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'Last-Modified'
19:15:30 | End exception report.
19:15:31 | request failed: 403 Client Error: Forbidden for url: https://cdn.gog.com/secure/agarest/extras/collectors_content/agarest_ce_digital_artbook.zip?1223fb43868c5f174d699acd5e3ffcf6201d8cfdec2a089be79108b742fa1f3738dd9e92b9f7fdf60dbd9ac9cb6342dbabe5da6aeb2bd0ca4c4416bdc0a687cbd93e9bfd5b9d681a4a1dfae7c1277154090f0975d8fd437ec94c881d9492946a899f19326de686947613e5357af3f2cba601b217c05c87e90ac8ae4f508312ce5c21542d1cf16fb&fileExtForIe=.exe. will not retry.
19:15:31 | failed to fetch https://www.gog.com/downlink/agarest_generations_of_war_collectors_edition_upgrade/38083
19:15:31 | Could not fetch file info so using canonical link: https://www.gog.com/downloads/agarest_generations_of_war_collectors_edition_upgrade/38083
19:15:31 | ( 5 / 3431) fetching game details for agarest_generations_of_war_2...
19:15:32 | failed to fetch https://www.gog.com/downloads/agarest_generations_of_war_2/58103 because of non-HTTP Error
19:15:32 | The handled exception was:
19:15:32 |
Traceback (most recent call last):
File "/home/user/bin/gog/gogrepoc/./gogrepoc.py", line 1244, in filter_extras
fetch_file_info(d, False,save_md5_xml,updateSession)
File "/home/user/bin/gog/gogrepoc/./gogrepoc.py", line 1088, in fetch_file_info
d.raw_updated = d.gog_data.headers["Last-Modified"]
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'Last-Modified'
19:15:32 | End exception report.
^C19:15:32 | exiting...

Is this reproducible, that the script throws an exception on trying to download metadata for my games, I think this is only if it tries to retrieve the data for the extras (manuals, wallpapers etc), not the installer itself.

It was mentioned that the relevant endpoint "downloads" was changed to "downlink", but I couldn't fully put my finger on it during debugging the script, could trace it back in the function "def filter_extras()" line 1195ff, but not sure how to solve it.

Can anyone confirm that, that the metadata loading of extras throws this exception?
Looks like you have an incomplete resume manifest, you should rollback and let the resume finish before updating to a new GOGrepo version ( I do check for this now but I'm guessing you're resume manifest old enough it's not versioned )

As a secondary issue GOG uses 403 kinda wrong in multiple places, it can mean server overloaded ( which should be a 500 error not a 400 one ) or that your session is invalid, so you should try logging in again using the login command.
Post edited January 18, 2025 by Kalanyr
Well, I did the nuclear option; removed the old installation, did a fresh git clone, a new login and then update:

I only censored the hash values of the DL urls and personal information.

It seems that there is a problem in gogrepoc, because I dont' t have a problem with lgogdownloader and gogg (the new in town).

$ ~/bin/gog> git clone https://github.com/Kalanyr/gogrepoc
Cloning into 'gogrepoc'...
remote: Enumerating objects: 504, done.
remote: Counting objects: 100% (135/135), done.
remote: Compressing objects: 100% (19/19), done.
remote: Total 504 (delta 119), reused 116 (delta 116), pack-reused 369 (from 1)
Receiving objects: 100% (504/504), 247.29 KiB | 3.34 MiB/s, done.
Resolving deltas: 100% (202/202), done.
$ ~/bin/gog> cd gogrepoc/
$ ~/bin/gog/gogrepoc> mcedit -d gogrepoc.py

$ ~/bin/gog/gogrepoc> ./gogrepoc.py login
Username: very.secret.shopping@example.com
Password:
16:26:44 | attempting Galaxy login as 'very.secret.shopping@example.com' ...
16:26:45 | saving token...
16:26:45 | saved token
16:26:45 | exiting...
$ ~/bin/gog/gogrepoc> ./gogrepoc.py update
16:26:53 | loading local manifest...
16:26:53 | loading token...
16:26:53 | loading local resume manifest...
16:26:53 | fetching game product data (page 1)...
16:26:55 | fetching game product data (page 2 / 35)...
...
16:27:39 | fetching game product data (page 35 / 35)...
16:27:40 | found 3482 games !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
16:27:40 | saving resume manifest...
16:27:41 | saved resume manifest
16:27:41 | ( 1 / 3482) fetching game details for 140_game...
16:27:42 | ( 2 / 3482) fetching game details for 1428_shadows_over_silesia...
16:27:43 | failed to fetch https://www.gog.com/downloads/1428_shadows_over_silesia_bonus_stories/94168 because of non-HTTP Error
16:27:43 | The handled exception was:
16:27:43 |
Traceback (most recent call last):
File "~/bin/gog/gogrepoc/./gogrepoc.py", line 1244, in filter_extras
fetch_file_info(d, False,save_md5_xml,updateSession)
File "~/bin/gog/gogrepoc/./gogrepoc.py", line 1088, in fetch_file_info
d.raw_updated = d.gog_data.headers["Last-Modified"]
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'Last-Modified'
16:27:43 | End exception report.
16:27:43 | request failed: 403 Client Error: Forbidden for url: https://cdn.gog.com/secure/1428_shadows_over_silesia_bonus_stories/1428_shadows_over_silesia_bonus_stories.zip?1223fb43868c5f174d699acf5f3ffcf657055d06f0acd6c5e2cf9203201871e3c009c4b2aa9173ddd8e5bfd0f12768f5f6410e6e2811d2de7088c325fe8c2a6488fc1f6345ff87890dc0a447692b728051784a0c4892fb7e31e20c2b3c2645179399eecbc77076ca1c38c2dacc9498c05c2f6374404ac3062f9f0061369bb75d489cf8751720f39f2947f56153b0cc30a7e591ea34cc998e07&fileExtForIe=.exe. will not retry.
16:27:43 | failed to fetch https://www.gog.com/downlink/1428_shadows_over_silesia_bonus_stories/94168
16:27:43 | Could not fetch file info so using canonical link: https://www.gog.com/downloads/1428_shadows_over_silesia_bonus_stories/94168
16:27:44 | failed to fetch https://www.gog.com/downloads/1428_shadows_over_silesia_soundtrack/94120 because of non-HTTP Error
16:27:44 | The handled exception was:
16:27:44 |
Traceback (most recent call last):
File "~/bin/gog/gogrepoc/./gogrepoc.py", line 1244, in filter_extras
fetch_file_info(d, False,save_md5_xml,updateSession)
File "~/bin/gog/gogrepoc/./gogrepoc.py", line 1088, in fetch_file_info
d.raw_updated = d.gog_data.headers["Last-Modified"]
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'Last-Modified'
16:27:44 | End exception report.
16:27:44 | request failed: 403 Client Error: Forbidden for url: https://cdn.gog.com/secure/1428_shadows_over_silesia_soundtrack/1428_shadows_over_silesia_soundtrack_mp3.zip?1223fb43868c5f174d699acf593ffcf6a5491c40e4cb456a2494ad9082d38fb4b30df34b6371a64d1db24bf7af9b723acc252ed4597ad77ce55a55c5858c7c784edf98e4d29b10ec91b6a639f8a5a1b27cebc69106cb8fd9fa141d5848341f2e2c50541c359f8974f3b49c4188bd86b79887e4c73a086530c866b4bab75c4c07f2b1c8a0859fbdf920a92600c6b5c15ce8b4aef07fc&fileExtForIe=.exe. will not retry.
16:27:44 | failed to fetch https://www.gog.com/downlink/1428_shadows_over_silesia_soundtrack/94120
16:27:44 | Could not fetch file info so using canonical link: https://www.gog.com/downloads/1428_shadows_over_silesia_soundtrack/94120
16:27:44 | ( 3 / 3482) fetching game details for 15_days...
16:27:45 | failed to fetch https://www.gog.com/downloads/15_days/79876 because of non-HTTP Error
16:27:45 | The handled exception was:
16:27:45 |
Traceback (most recent call last):
File "~/bin/gog/gogrepoc/./gogrepoc.py", line 1244, in filter_extras
fetch_file_info(d, False,save_md5_xml,updateSession)
File "~/bin/gog/gogrepoc/./gogrepoc.py", line 1088, in fetch_file_info
d.raw_updated = d.gog_data.headers["Last-Modified"]
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'Last-Modified'
16:27:45 | End exception report.
^C16:27:45 | exiting...
Post edited January 18, 2025 by coffeecup