It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4034 posts)(4034 posts)
(4034 posts)
Pages:
1
50
100
150
200
250
266
267
268
269
This is my favourite topic
Posted December 05, 2024
avatar
kenrap: It's how it's executed with the shell script on Linux as if it were a binary.
Okay, so it's a check in Linux. That's what I wasn't clear on, but suspected.

avatar
kenrap: You'd have to open up the shell script with a text editor and remove all the shell script code up to line 519 and change the file extension as "tar.gz" since by then it will become an archive file with the binary data.
Thanks for the info.
To keep things simple in Windows, I guess I will just continue to use 7-Zip with GOG's Linux files.
Hopefully that will be enough of a check for corruption etc.

avatar
joveian: The installer itself is MojoSetup, although it doesn't seem to have an equivalent of innoextract. I see you are right though that the rest of the file is a zip file (I guess the test is just CRC32 though). The lua config for MojoSetup is inside that zip file.

Personally, I think Itch has the right idea and zip files are good for distribution and have the app (Galaxy for GOG) deal with anything more complex. The Itch build tool is open source and they are inviting others to use the same methods; I wish GOG would do so.
Thanks for the info.

Last time I checked an Itch.io download, I seem to recall they were also using InnoSetup for their Windows installers.

avatar
joveian: (not to hijack this thread with any more discussion of this topic though).
Yeah, we better nip it in the bud.
Post edited December 05, 2024 by Timboli
Posted December 15, 2024
Just a small bug report: executing "clean" command while \!orphaned\!md5_xmls\ path exists crashes the script. Removing / renaming this folder solves the issue.
Post edited December 15, 2024 by KoЯni
Posted December 17, 2024
avatar
KoЯni: Just a small bug report: executing "clean" command while \!orphaned\!md5_xmls\ path exists crashes the script. Removing / renaming this folder solves the issue.
*blink* How did you manage to get !md5_xmls in !orphaned ? What is your directory structure ? The only structure that I can think of that would get !md5_xmls into !orphaned would nuke the script itself and the login token every time its run.

ETA - !md5_xmls aren't orphanable by the script because there'd be no point, if the MD5 chunk data is outdate you can't actually use it for anything because the file you'd need to access to repair it is pretty much definitionally not available
Post edited December 17, 2024 by Kalanyr
Posted December 18, 2024
Hi Kalanyr,

Here's my folder structure. The script sits happily in the root of the library:
gog
|
---!orphaned
---!downloading
---game1folder
...
gogrepoc.py

I run the following sequence of commands to update my repo. It might be worth to note that I'm not using any paths, except the current dir dot for the "clean" command (it doesn't accept omitting the path entirely):
python gogrepoc.py update -lang en pl -os windows -full -md5xmls
python gogrepoc.py download -covers -backgrounds
python gogrepoc.py clean .
python gogrepoc.py verify

1. The first command adds !md5_xmls folder in the gog root directory with corresponding XML files:
gog
|
---!orphaned
---!downloading
---!md5_xmls
|
---game1\downloads\windows\English\game1exe.xml
---game1\downloads\windows\English\game1bin.xml
...
---game1folder
...
gogrepoc.py

This is reflected by "Handling MD5 XML info for $game" in the log.

Command (2) just downloads stuff into !downloading folder, after cleaning it (and the !provisional subfolder). Nothing out of ordinary here.

Command (3) orphanes (among other files) the !md5_xmls folder, moving it to !orphaned folder. From the log:
00:22:13 | loading local manifest...
00:22:20 | scanning local directories within '.'...
00:22:20 | scanning manifest for renames...
00:22:20 | orphaning dir '!md5_xmls'

But this is done only if !md5_xmls has not already been orphaned in the previous run. This is what happens otherwise:
00:26:40 | loading local manifest...
00:26:46 | scanning local directories within '.'...
00:26:46 | scanning manifest for renames...
00:26:46 | orphaning dir '!md5_xmls'
00:26:46 | fatal...
Traceback (most recent call last):
File "gogrepoc.py", line 3980, in <module>
main(process_argv(sys.argv))
File "gogrepoc.py", line 3800, in main
cmd_clean(args.cleandir, args.dryrun)
File "gogrepoc.py", line 3562, in cmd_clean
move_with_increment_on_clash(cur_fulldir, os.path.join(orphan_root_dir,cur_dir))
File "gogrepoc.py", line 469, in move_with_increment_on_clash
move_with_increment_on_clash(src,dst,count+1)
File "gogrepoc.py", line 449, in move_with_increment_on_clash
root,ext = path_preserving_split_ext(dst)
File "gogrepoc.py", line 442, in path_preserving_split_ext
extension = os.extsep + os.path.basename(path).rsplit(os.extsep,1)[1]
IndexError: list index out of range

Finally, the VERIFY command - again, nothing unexpected here.

My guess is that the issue is in handling the dot as current dir path in the CLEAN command or maybe the !md5_xmls folder should not be orphaned in the first place.

Happy to provide any more details if you need it for your great tool, for which I'm very thankful :)
EDIT: I see spaces has been removed, I hope you can still read the folder structure.
Post edited December 18, 2024 by KoЯni
Posted December 18, 2024
avatar
KoЯni: Hi Kalanyr,

Here's my folder structure. The script sits happily in the root of the library:
gog
|
---!orphaned
---!downloading
---game1folder
...
gogrepoc.py

I run the following sequence of commands to update my repo. It might be worth to note that I'm not using any paths, except the current dir dot for the "clean" command (it doesn't accept omitting the path entirely):
python gogrepoc.py update -lang en pl -os windows -full -md5xmls
python gogrepoc.py download -covers -backgrounds
python gogrepoc.py clean .
python gogrepoc.py verify

1. The first command adds !md5_xmls folder in the gog root directory with corresponding XML files:
gog
|
---!orphaned
---!downloading
---!md5_xmls
|
---game1\downloads\windows\English\game1exe.xml
---game1\downloads\windows\English\game1bin.xml
...
---game1folder
...
gogrepoc.py

This is reflected by "Handling MD5 XML info for $game" in the log.

Command (2) just downloads stuff into !downloading folder, after cleaning it (and the !provisional subfolder). Nothing out of ordinary here.

Command (3) orphanes (among other files) the !md5_xmls folder, moving it to !orphaned folder. From the log:
00:22:13 | loading local manifest...
00:22:20 | scanning local directories within '.'...
00:22:20 | scanning manifest for renames...
00:22:20 | orphaning dir '!md5_xmls'

But this is done only if !md5_xmls has not already been orphaned in the previous run. This is what happens otherwise:
00:26:40 | loading local manifest...
00:26:46 | scanning local directories within '.'...
00:26:46 | scanning manifest for renames...
00:26:46 | orphaning dir '!md5_xmls'
00:26:46 | fatal...
Traceback (most recent call last):
File "gogrepoc.py", line 3980, in <module>
main(process_argv(sys.argv))
File "gogrepoc.py", line 3800, in main
cmd_clean(args.cleandir, args.dryrun)
File "gogrepoc.py", line 3562, in cmd_clean
move_with_increment_on_clash(cur_fulldir, os.path.join(orphan_root_dir,cur_dir))
File "gogrepoc.py", line 469, in move_with_increment_on_clash
move_with_increment_on_clash(src,dst,count+1)
File "gogrepoc.py", line 449, in move_with_increment_on_clash
root,ext = path_preserving_split_ext(dst)
File "gogrepoc.py", line 442, in path_preserving_split_ext
extension = os.extsep + os.path.basename(path).rsplit(os.extsep,1)[1]
IndexError: list index out of range

Finally, the VERIFY command - again, nothing unexpected here.

My guess is that the issue is in handling the dot as current dir path in the CLEAN command or maybe the !md5_xmls folder should not be orphaned in the first place.

Happy to provide any more details if you need it for your great tool, for which I'm very thankful :)
EDIT: I see spaces has been removed, I hope you can still read the folder structure.
I see. That's definitely not intended but it should be easy enough to patch it so it keeps working for you.

In the mean time find
ORPHAN_DIR_EXCLUDE_LIST = [ORPHAN_DIR_NAME,DOWNLOADING_DIR_NAME,IMAGES_DIR_NAME, '!misc']

and change it too

ORPHAN_DIR_EXCLUDE_LIST = [ORPHAN_DIR_NAME,DOWNLOADING_DIR_NAME,IMAGES_DIR_NAME, '!misc','!md5_xmls']

then delete !md5_xmls from !orphaned and it should go back to the prior behaviour.

I might have to add some code that's a little more rigorous about validating the closeness to intendedness of the directory structure in future but I guess if it works for now it's fine.
Posted December 18, 2024
Thanks Kalanyr, this is just a minor issue (I manually review orphaned files anyway before deleting) and the fix is easy and effective enough - not sure if any additional work needs to be done here. Thanks again!
Posted December 25, 2024
Just unpacked my new 4TB SSD and am in the process of filling ~80% of its capacity with my GOG owned games using the script. 2990 GB to go. Definitely wouldn't want to be downloading the installers + extras for the 687 games I own manually.

So just a word of thanks. Worked like a charm.
Post edited December 25, 2024 by idbeholdME
Posted December 27, 2024
Are game downloads supposed to go into the provisional folder now? Why do I have so many duplicate directories. My download command is
python "F:\Files\GoGRepo\gogrepoc.py" download -skipextras "F:\Files\Game Downloads\GoG Downloads" >>"F:\Files\GoGRepo\Logs\Download\GoGRepo_Download_Log_%date:~10%%date:~4,2%%date:~7,2%.txt"
Posted December 27, 2024
avatar
paladin181: Are game downloads supposed to go into the provisional folder now? Why do I have so many duplicate directories. My download command is
python "F:\Files\GoGRepo\gogrepoc.py" download -skipextras "F:\Files\Game Downloads\GoG Downloads" >>"F:\Files\GoGRepo\Logs\Download\GoGRepo_Download_Log_%date:~10%%date:~4,2%%date:~7,2%.txt"
Files are downloaded to !provisional, moved (one file at a time) to !downloading whenever a single file download has been completed, and then moved (in bulk) to their final resting place when the entire operation is done. During the download you will have the same game folders in !provisional and !downloading, but once the entire download is done, all game folders in both should be gone. That is my experience. Do you see something different?

EDIT: correction:
Files are downloaded to !downloading, moved (one file at a time) to !provisional whenever a single file download has been completed, and then moved (in bulk) to their final resting place when the entire operation is done.
Post edited December 28, 2024 by mrkgnao
Posted December 28, 2024
avatar
mrkgnao: Files are downloaded to !provisional, moved (one file at a time) to !downloading whenever a single file download has been completed, and then moved (in bulk) to their final resting place when the entire operation is done. During the download you will have the same game folders in !provisional and !downloading, but once the entire download is done, all game folders in both should be gone. That is my experience. Do you see something different?
Well, I'm downloading 6TB of files over several sessions. So the files are in downloading, get moved to provisional when the file is done, and sit there (their status is listed as working in my log files). I don't know what that means. But they aren't moved to the final location yet.

Here is the log entry of a completed file:
07:58:14 | moving provisionally completed download 'F:\Files\Game Downloads\GoG Downloads\!downloading\chronos_before_the_ashes\setup_chronos_before_the_ashes_1.1_(44399)-2.bin' to 'F:\Files\Game Downloads\GoG Downloads\!downloading\!provisional\chronos_before_the_ashes\setup_chronos_before_the_ashes_1.1_(44399)-2.bin'

And here is one that was moved to provisional in a previous session:
01:02:38 | working setup_1428_shadows_over_silesia_1.1.34_(64bit)_(78513).exe
Post edited December 28, 2024 by paladin181
Posted December 28, 2024
avatar
mrkgnao: Files are downloaded to !provisional, moved (one file at a time) to !downloading whenever a single file download has been completed, and then moved (in bulk) to their final resting place when the entire operation is done. During the download you will have the same game folders in !provisional and !downloading, but once the entire download is done, all game folders in both should be gone. That is my experience. Do you see something different?
avatar
paladin181: Well, I'm downloading 6TB of files over several sessions. So the files are in downloading, get moved to provisional when the file is done, and sit there (their status is listed as working in my log files). I don't know what that means. But they aren't moved to the final location yet.

Here is the log entry of a completed file:
07:58:14 | moving provisionally completed download 'F:\Files\Game Downloads\GoG Downloads\!downloading\chronos_before_the_ashes\setup_chronos_before_the_ashes_1.1_(44399)-2.bin' to 'F:\Files\Game Downloads\GoG Downloads\!downloading\!provisional\chronos_before_the_ashes\setup_chronos_before_the_ashes_1.1_(44399)-2.bin'

And here is one that was moved to provisional in a previous session:
01:02:38 | working setup_1428_shadows_over_silesia_1.1.34_(64bit)_(78513).exe
Sorry. I messed things up.

Correction:
Files are downloaded to !downloading, moved (one file at a time) to !provisional whenever a single file download has been completed, and then moved (in bulk) to their final resting place when the entire operation is done.

When the entire download operation is over, !provisional and !downloading should be empty. Is that not the case?
Post edited December 28, 2024 by mrkgnao
Posted December 28, 2024
avatar
mrkgnao: Correction:
Files are downloaded to !downloading, moved (one file at a time) to !provisional whenever a single file download has been completed, and then moved (in bulk) to their final resting place when the entire operation is done.

When the entire download operation is over, !provisional and !downloading should be empty. Is that not the case?
I wasn't aware of this either, unless it is a very recent change. I thought incomplete files are in the !downloading and are moved to the main download directory when they are complete (each file one by one).

Any idea why there is now the extra step and directory, !provisional?
Posted December 28, 2024
avatar
mrkgnao: Sorry. I messed things up.

Correction:
Files are downloaded to !downloading, moved (one file at a time) to !provisional whenever a single file download has been completed, and then moved (in bulk) to their final resting place when the entire operation is done.

When the entire download operation is over, !provisional and !downloading should be empty. Is that not the case?
Haven't properly completed the downloading portion. It's 6TB, and I just started 26.12. So it will take a while, I'm only downloading for about 7 or 8 hours a day, I've gotten through the first TB, or close to it.
Posted December 28, 2024
avatar
timppu: I wasn't aware of this either, unless it is a very recent change. I thought incomplete files are in the !downloading and are moved to the main download directory when they are complete (each file one by one).

Any idea why there is now the extra step and directory, !provisional?
It's fairly new. You will have to ask Kalanyr for the reason.

avatar
paladin181: Haven't properly completed the downloading portion. It's 6TB, and I just started 26.12. So it will take a while, I'm only downloading for about 7 or 8 hours a day, I've gotten through the first TB, or close to it.
I expect you should be ok then. How do you stop a download in the middle?
Post edited December 28, 2024 by mrkgnao
Posted December 29, 2024
avatar
mrkgnao: It's fairly new. You will have to ask Kalanyr for the reason.

I expect you should be ok then. How do you stop a download in the middle?
I close the cmd window.
Pages:
1
50
100
150
200
250
266
267
268
269
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4034 posts)(4034 posts)
(4034 posts)