It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4238 posts)(4238 posts)
(4238 posts)
Pages:
1
50
100
150
187
188
189
190
191
250
283
This is my favourite topic
Posted December 13, 2021
Yes. When I tried to run update it was saying nothing found but only a little more than half my games are downloaded.

Easiest proof of what Im saying is that I started at 11pm last night approx. Its still running now. Not 1 megabyte of data more has been used since i started and yes I have tried refreshing to make sure. I even closed and reopened and its reading the same. Its not making copies but when it gets to that spot it seems to delete the bin files and redownload them.
Post edited December 13, 2021 by eatatralphs
Posted December 13, 2021
avatar
eatatralphs: Yes. When I tried to run update it was saying nothing found but only a little more than half my games are downloaded.

Easiest proof of what Im saying is that I started at 11pm last night approx. Its still running now. Not 1 megabyte of data more has been used since i started and yes I have tried refreshing to make sure. I even closed and reopened and its reading the same. Its not making copies but when it gets to that spot it seems to delete the bin files and redownload them.
If you look at gogrepo.log, what does it say when it should be moving a completed file to the target directory, e.g.:

22:56:30 | moving completed download '\gog\!downloading\cyberpunk_2077_game\setup_cyberpunk_2077_build_3551602change_4457769_galaxydll_(64bit)_(500 01)-25.bin' to '\gog\cyberpunk_2077_game\setup_cyberpunk_2077_build_3551602change_4457769_galaxydll_(64bit)_(50001)-25.bin'
Because your description sounded like that part somehow fails, it fails to move the completed file from the !downloading directory to the target directory?

No idea what could be causing your problem, but since you had some problem with the filesystem getting full on your earlier try, I would probably try these things:

1. Check your filesystem (Error Checking/chkdsk), both where you run the gogrepo script, and where the target directory is.

2. Run the gogrepo "clean" command to the target (download) directory, just in case there are some obsolete files. They will be moved to the "!orphaned"-directory.

3. Delete the unfinished downloads, if any, that are inside the "!downloading" directory.

4. Delete (or rename) the existing gog-manifest.dat file, and run gogrepo update again, in order to create the whole manifest file from scratch. This is time-consuming and possibly unnecessary if running "update" currently doesn't report any problems with the existing manifest file, but you never know...

5. Run gogrepo download again.

And you are using the Kalanyr branch version of the gogrepo, right? E.g. this one:

https://github.com/Kalanyr/gogrepoc

Still one question, just to be sure:

avatar
eatatralphs: My download though was interrupted due to my drive getting full. I have cleared another terrabytre off but when I tried to resume it is now wanting to redownload everything from start.
When you say you "cleared" some space in the target directory, you didn't touch any of the already downloaded GOG game installers, right? If you e.g. moved some of your already downloaded files to some other place, then gogrepo will try to redownload them because it doesn't find them in the target directory anymore.

I needed to ask that just so that there is no misconception that gogrepo would somehow remember what you have downloaded already, even if you move the downloaded files to some other place. I think the script always checks in the beginning of the download session which files, which it can see in the manifest file, it can find in the target directory (checking the filenames and sizes for all files). If some files are missing because you moved them, it will consider them missing and redownload them.
Post edited December 13, 2021 by timppu
Posted December 14, 2021
avatar
eatatralphs: Yes. When I tried to run update it was saying nothing found but only a little more than half my games are downloaded.

Easiest proof of what Im saying is that I started at 11pm last night approx. Its still running now. Not 1 megabyte of data more has been used since i started and yes I have tried refreshing to make sure. I even closed and reopened and its reading the same. Its not making copies but when it gets to that spot it seems to delete the bin files and redownload them.
It will delete bins and redownload them if they don't match what they should according to the manifest. It should be capable of recovering partial files eg interuppted by running out of space, but that hasn't really been tested, since it shouldn't occur normally (corrupt files shouldn't leave !downloading for the most part these days).

If you have a look in the log files and there is a specific file / folder it's getting stuck on you could try deleting that file/folder.

Also to be clear the order to update your current files is to do: update (which updates the manifest), download (which downloads the files in the manifest if they are different/missing), verify (which will check that the files are correct) , you can add in a clean wherever you want (except before update) it just tidies stuff up and makes sures the folder structure is correct and there's no extra files. You can also use trash to get rid of orphaned files if you're sure you don't want them any more.

ETA - Another possibility is that GOG update the thing you're trying to download without making it as updated in which you can use the -ids flag on the update command to specify game(s) to force a sync with the server
Post edited December 14, 2021 by Kalanyr
Posted December 16, 2021
As a headsup I recommend avoiding updating Imperator Rome if possible, there appears to be a synchronization issue on GOG servers and your chance of having all the files be available (and thus getting a correct manifest) is <10%. If you're doing a full update its probably best to specifically skip it. All the files *are* available if you're persistent and want to download them manually from the website, you just may have to try different browsers and times of day to hit a server that actually has the file you want available.
Posted December 18, 2021
2 heads up

1) It's the Winter Sale, this is a terrible time to do Full Manifest Updates or create a manifest. Hold off until the 2nd week or so of January to do those things if you can

2) This one probably has less practical effect, GOG seems to be having issues serving the 32-bit Windows installer for Mechajammer. Expect this download to fail a lot. Seems to be a similar issue to Imperator because it is available on some of their servers.
Posted December 18, 2021
I used gogrepo 2 years ago. My old script don't work.
Which is the current, working version?
Thanks!
Posted December 18, 2021
avatar
Lebostein: I used gogrepo 2 years ago. My old script don't work.
Which is the current, working version?
Thanks!
Did you try downloading the latest version of either gogrepo, gogrepoc or the UI implementation of gogrepo?
And do you have Python 3 installed? (I don't think the script supports Python or Python 2, so mind that)
And make sure your system recognizes commands that use Python (you might need to set environment paths if you're on Windows, or do a purge and fresh install if you're on Linux)
Also, if you are on Windows and tried running the script with PowerShell, try to run it from the CMD. And if you tried on CMD, try on PowerShell instead.
And did you try installing what the repository asks with pip? As far as I know, on CMD, the pip command is a bit different, where you need to use "py -m pip" instead of just "pip" (so you need to use the command py -m pip install html5lib html2text to install gogrepo's dependencies)
And lastly, make sure you are using the right commands for your OS. Android with Termux should accept commands starting with either python or python3, Linux seems to only accept Python commands with python3, and, if my notes where I checked the pip info above are right, Windows should use just py.
Posted December 18, 2021
avatar
_Auster_: And do you have Python 3 installed? (I don't think the script supports Python or Python 2, so mind that)
The script (gogrepoc) works just fine with Python 2.7.
Posted December 18, 2021
avatar
mrkgnao: The script (gogrepoc) works just fine with Python 2.7.
Thanks for confirming!
Posted December 19, 2021
avatar
Lebostein: I used gogrepo 2 years ago. My old script don't work.
Which is the current, working version?
Thanks!
avatar
_Auster_: Did you try downloading the latest version of either gogrepo, gogrepoc or the UI implementation of gogrepo?
And do you have Python 3 installed? (I don't think the script supports Python or Python 2, so mind that)
And make sure your system recognizes commands that use Python (you might need to set environment paths if you're on Windows, or do a purge and fresh install if you're on Linux)
Also, if you are on Windows and tried running the script with PowerShell, try to run it from the CMD. And if you tried on CMD, try on PowerShell instead.
And did you try installing what the repository asks with pip? As far as I know, on CMD, the pip command is a bit different, where you need to use "py -m pip" instead of just "pip" (so you need to use the command py -m pip install html5lib html2text to install gogrepo's dependencies)
And lastly, make sure you are using the right commands for your OS. Android with Termux should accept commands starting with either python or python3, Linux seems to only accept Python commands with python3, and, if my notes where I checked the pip info above are right, Windows should use just py.
Mac OS + Python3 + gogrepoc:
ModuleNotFoundError: No module named 'CoreFoundation'
Posted December 20, 2021
avatar
_Auster_: Did you try downloading the latest version of either gogrepo, gogrepoc or the UI implementation of gogrepo?
And do you have Python 3 installed? (I don't think the script supports Python or Python 2, so mind that)
And make sure your system recognizes commands that use Python (you might need to set environment paths if you're on Windows, or do a purge and fresh install if you're on Linux)
Also, if you are on Windows and tried running the script with PowerShell, try to run it from the CMD. And if you tried on CMD, try on PowerShell instead.
And did you try installing what the repository asks with pip? As far as I know, on CMD, the pip command is a bit different, where you need to use "py -m pip" instead of just "pip" (so you need to use the command py -m pip install html5lib html2text to install gogrepo's dependencies)
And lastly, make sure you are using the right commands for your OS. Android with Termux should accept commands starting with either python or python3, Linux seems to only accept Python commands with python3, and, if my notes where I checked the pip info above are right, Windows should use just py.
avatar
Lebostein: Mac OS + Python3 + gogrepoc:
ModuleNotFoundError: No module named 'CoreFoundation'
Big Sur changed how system libraries are accessed, this should be fixed with Python 3.7 (though I'd recommend the latest or at least 3.8 if possible (3.7 was at security updates only before this was fully resolved))
avatar
_Auster_: Did you try downloading the latest version of either gogrepo, gogrepoc or the UI implementation of gogrepo?
And do you have Python 3 installed? (I don't think the script supports Python or Python 2, so mind that)
And make sure your system recognizes commands that use Python (you might need to set environment paths if you're on Windows, or do a purge and fresh install if you're on Linux)
Also, if you are on Windows and tried running the script with PowerShell, try to run it from the CMD. And if you tried on CMD, try on PowerShell instead.
And did you try installing what the repository asks with pip? As far as I know, on CMD, the pip command is a bit different, where you need to use "py -m pip" instead of just "pip" (so you need to use the command py -m pip install html5lib html2text to install gogrepo's dependencies)
And lastly, make sure you are using the right commands for your OS. Android with Termux should accept commands starting with either python or python3, Linux seems to only accept Python commands with python3, and, if my notes where I checked the pip info above are right, Windows should use just py.
avatar
Lebostein: Mac OS + Python3 + gogrepoc:
ModuleNotFoundError: No module named 'CoreFoundation'
Big Sur changed how system libraries are accessed, this should be fixed with Python 3.7+ (though I'd recommend the latest or at least 3.8+ if possible (3.7 was at security updates only before this was fully resolved))
Post edited December 20, 2021 by Kalanyr
Posted December 20, 2021
Mac OS 10.13.6 with Python 3.9.9
Posted December 21, 2021
avatar
Lebostein: Mac OS 10.13.6 with Python 3.9.9
Try installing this: https://pypi.org/project/pyobjc/
Posted December 25, 2021
avatar
Kalanyr: It should be capable of recovering partial files eg interuppted by running out of space, but that hasn't really been tested, since it shouldn't occur normally (corrupt files shouldn't leave !downloading for the most part these days).
I know that with the no delete change I made that it does indeed continue where it left off.
Posted January 02, 2022
I have troubles with gogrepoc (kalanyr version)

I had a 1 year old repository and wanted to download the new games I purchased (amongst them: cyberpunk 2077, and many others)

I installed python 3.10 (new computer) and redownloaded today the gogrepoc from kalanyr github

I then did:

E:\__games__\gog>python gogrepoc\gogrepoc.py verify -os windows linux -lang en fr -skipmd5 -skipzip ici

E:\__games__\gog>python gogrepoc\gogrepoc.py download -os "windows,linux" -lang "'en', 'fr'" -skipgalaxy ici

E:\__games__\gog>python gogrepoc\gogrepoc.py login

E:\__games__\gog>python gogrepoc\gogrepoc.py update

E:\__games__\gog>python gogrepoc\gogrepoc.py download -os windows linux -lang en fr -skipgalaxy -skipextras ici

== notices many (all?) ici\gamename folders didn't contain the binaries!

E:\__games__\gog>python gogrepoc\gogrepoc.py update -full

E:\__games__\gog>python gogrepoc\gogrepoc.py verify -os windows linux -lang en fr -skipmd5 ici

E:\__games__\gog>python gogrepoc\gogrepoc.py download -os windows linux -lang en fr -skipgalaxy ici

== still do not contain the binaries (but contains extras)

I am at a loss: why doesn't it download binaries? is it because of "skipgalaxy" (I want the binaries, and not the galaxy ones, as I don't like galaxy) ?

And why does "verify" do not notice that the binaries are missing??

If I don't specify "-skipgalaxy", nothing new is downloaded
And during the first pass (when it verifies existing things) I see lots (lots!) of game dirs without binaries (but just with extras), and a few with binaries (not sure if they are recent or not....)

for exemple:

latest try:
E:\__games__\gog>python gogrepoc\gogrepoc.py download -os windows linux -lang en fr -- ici
it went very quickly and didn't download a things (I dropped "-skipgalaxy" and expected many things to be added...)

it shows only the verification phase, such as (extract) :

... hundreds of lines (I have >600 games) ....
04:39:11 | pass cultures_2_ost.zip
04:39:11 | {cyberpunk_2077_game} ## no install binaries! just extas! game is missing!
04:39:12 | pass wallpapers_en.zip
04:39:12 | pass wallpapers_kr.zip
04:39:12 | pass wallpapers_zhs.zip
04:39:12 | pass wallpapers_zht.zip
04:39:12 | pass additional_wallpapers_en.zip
04:39:12 | pass additional_wallpapers_kr.zip
04:39:12 | pass additional_wallpapers_zhs.zip
04:39:12 | pass additional_wallpapers_zht.zip
04:39:12 | pass cyberpunk_2077_mobile_wallpapers.zip
04:39:12 | pass cyberpunk_2077_sourcebook_en.zip
04:39:12 | pass cyberpunk_2077_sourcebook_pl.zip
04:39:12 | pass cyberpunk_2077_ost_wav.zip
04:39:12 | pass cyberpunk_2077_ost_mp3.zip
04:39:12 | pass cyberpunk_2077_ost_flac.zip
04:39:12 | pass cyberpunk_2077_avatars.zip
04:39:12 | pass cyberpunk_2077_big_city_dreams.zip
04:39:12 | pass cyberpunk_2077_posters.zip
04:39:12 | {dark_fall_the_journal} ## no install binaries! just extas! game is missing!
04:39:12 | pass dark_fall_manual.zip
04:39:12 | pass dark_fall_soundtrack.zip
04:39:12 | {darklands} ## no install binaries! just extas! game is missing!
04:39:12 | pass darklands_manual1.zip
04:39:12 | pass darklands_cluebook1.zip
04:39:12 | pass darklands_reference_card.zip
04:39:12 | pass darklands_soundtrack.zip
04:39:12 | pass darklands_ost_flac.zip
04:39:12 | pass darklands_map.zip
04:39:12 | {darkstone}
04:39:12 | pass setup_darkstone_1.0.5b_(french)_(24324).exe ### a binary !
04:39:12 | pass darkstone_manual.zip
04:39:12 | pass darkstone_wallpapers.zip
04:39:12 | pass darkstone_soundtrack.zip
... hundreds of lines (I have >600 games) ....

many, many games dir are missing .exe, .bin, .tar and the likes!

list of game dirs without a setup*.exe:

$ n=0; for i in [a-zA-Z]*/; do ((n++)); ( cd "$i" && ls -l setup*.exe >/dev/null 2>&1 ) || echo "$i" ; done ; echo "verified $n game dirs"
anachronox/
baldurs_gate_iii/
becastled_demo/
blackhole/
blood_omen_legacy_of_kain/
call_to_power_2-extras/
carcassonne_tiles_tactics/
crusader_no_remorse/
cyberpunk_2077_game/
deus_ex_human_revolution_directors_cut/
deus_ex_mankind_divided/
dishonored_definitive_edition/
homeworld_deserts_of_kharak/
homeworld_emergence/
homeworld_remastered_collection/
kingdom_come_deliverance/
lost_horizon_2/
mark_of_the_ninja_remastered/
monkey_island-extras/
moto_racer_2/
neighbours_back_from_hell_game/
pillars_of_eternity/
pillars_of_eternity_2_game/
rebel_galaxy_outlaw/
return_of_the_incredible_machine_contraptions/
secret_files_3/
secret_files_sam_peters/
sherlock_holmes_crimes_and_punishments/
simcity_3000/
state_of_mind/
styx_master_of_shadows/
surviving_mars/
the_even_more_incredible_machine/
the_great_perhaps/
the_incredible_machine_3/
the_incredible_machine_even_more_contraptions/
ticket_to_ride/
venetica/
victor_vran/
war_wind_ii_human_onslaught/
warlords/
warlords_ii_deluxe/
warlords_iii_darklords_rising/
wizard_of_legend/
xcom2-extras/
verified 609 game dirs
Post edited January 02, 2022 by EdhelDil
Pages:
1
50
100
150
187
188
189
190
191
250
283
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4238 posts)(4238 posts)
(4238 posts)