It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3862 posts)(3862 posts)
(3862 posts)
Pages:
1
30
31
32
33
34
50
100
150
200
250
258
This is my favourite topic
Posted October 22, 2016
I have created a Windows Powershell script for myself to simplify the process of running gogrepo.py. This script gives me the option of either running a complete or incremental update of my library. Now, I'm no programmer, as will be apparent by the quality of my script, but it works for me and so I thought I would share it in case someone finds it helpful or wants to use this as a building block to something better or more ambitious.

The script will always "clean" the library but will only "verify" it if something changes in the library during the clean or download processes.

I have included an option to shut the computer down at the end of the entire update process.

The script can be downloaded from here: [url=]https://1drv.ms/u/s!AkusSFHlKO4LmjlvAEgA6nY_HNAY[/url]

At the moment it is locked to my library location, i.e. 'S:\Games\GOG Games', Windows and English updates as well as to including both games AND goodies. These can easily be changed by editing the appropriate lines in the script.

It also assumes that Python is installed and located in 'c:\python27'. This can be changed by editing the appropriate lines in the script file.

To use, extract the script to the same folder as gogrepo.py, right-click on the "gogrepo Powershell.ps1" script file and select "Run with Powershell".
Post edited October 23, 2016 by ikrananka
Posted October 23, 2016
As a user, to my understanding (the author can clarify):

avatar
phaolo: 1- does Gogrepo allow speed limits? (to avoid saturating the line)
2- does it have an option to filter downloads? E.g: game added to gog >01\2015 or updated >09\2016. Or check updates only for TW2 and TW3, etc..
(in theory, some integration with Magog to use its advanced search results would be quite cool!)
3- can it download files directly to an external hdd?
4- can it split games to a folder and extras to another?
5- can it pre-allocate files to avoid fragmentation?
6- renaming folders\files will break Gogrepo's capatibility to update installers, right? :\
1. No. If you need most of your internet bandwidth for something else (like watching Netflix), then either abort the gogrepo downloads with Ctrl-C, and continue later by running the download command again. Or find some third party solution to limit the bandwidth of certain PC (where gogrepo is running) or certain application (gogrepo).

From my experience though, I can use my internet pretty ok even though gogrepo is downloading, as long as I don't try to play some online game. Normal web browsing etc. is still ok and even watching youtube videos, so other apps will "steal" bandwidth back from gogrepo as needed.

2. There is an -id option for the various commands, at least for the update and download, and I think verify, commands. So you can filter it to one specific title (if you want to e.g. download, update or verify only one GOG game with gogrepo). I am not aware of any other more advanced filtering methods like downloading only games starting with certain alphabet or based on when they were released. Of course you can limit update based on OS (windows, linux and/or mac) and language versions.

3. Yes, I do that because my USB hard drive is the only HDD big enough to keep all my GOG (English Windows) games. You can specify the drive and/or path for download and verify commands, and for the clean command the path is even required (so that you don't accidentally run "clean" for your Windows directory, for example :)).

4. You can tell the download command to only download games or extras (with the -skipextras and -skipgames options), so I guess yes, by running the download command twice to two different paths.

5. I don't think so, but I wouldn't think fragmentation is a big problem when using gogrepo. After all, the files are mostly quite big, and GOG downloads them sequentially if you wish (tell the script to download only one file at a time; by default it downloads four files at the same time I think).

6. No, gogrepo will simply consider wrongly named files/folders as "obsolete". So the clean command will move them to the !orphaned folder, and the download command will download the files again (with correct names). So for example when GOG changed the folder name for The Witcher 3 files, gogrepo did download all the TW3 files all over again, unless you manually changed the folder name to the new one (then gogrepo would recognize them).

Note about the terns with gogrepo: "update" is the process of getting file details from GOG to your local manifest file, while "download" downloads any changed or new files according to that manifest file, by comparing it to what GOG game installers/extras you have in your hard drive already.

So it isn't like you run the "gogrepo update" to update your GOG games, or "gogrepo download" to download only new games. You use both commands every time you want to download anything (new or updated) from GOG.
Post edited October 23, 2016 by timppu
Posted October 23, 2016
Thanks for the Windows script, I'll also take a look into it. I've made only a bash script for Linux for myself, but I was less familiar with Windows Powershell scripting, so I think I'll just change your script for my purposes. :)

avatar
ikrananka: The script will always "clean" the library but will only "verify" it if something changes in the library during the clean or download processes.
One thing I'd like for gogrepo is that it would keep track which files (or titles) it actually downloads, and then verify only those. It is just that full verification takes a very long time for me (usually the whole weekend for 1340 games), so I'd prefer normally verifying only recently downloaded games (or files).

For now I do it so that I track manually which games gogrepo downloads (either as new or updated games), and then I run verify for them one by one, by using the -id option.

EDIT: Now that I think of it, maybe it would be possible to do something like that myself in Linux. Grep from the gogrepo download command scan only those titles which it says will be downloaded, and then afterwards tell verify to check only those titles one by one, with the -id option...
Post edited October 23, 2016 by timppu
Posted October 23, 2016
avatar
timppu: 1. [..] abort the gogrepo downloads with Ctrl-C, and continue later by running the download command again. [..]

2. [..] you can filter it to one specific title [..] I am not aware of any other more advanced filtering methods [..] Of course you can limit update based on OS (windows, linux and/or mac) and language versions.

3. [..] You can specify the drive and/or path for download and verify commands, and for the clean command the path is even required (so that you don't accidentally run "clean" for your Windows directory, for example :)).

4. You can tell the download command to only download games or extras (with the -skipextras and -skipgames options), so I guess yes, by running the download command twice to two different paths.

5. I don't think so, but I wouldn't think fragmentation is a big problem when using gogrepo. [..]

6. No, gogrepo will simply consider wrongly named files/folders as "obsolete". [..] unless you manually changed the folder name to the new one (then gogrepo would recognize them). [..]
So it isn't like you run the "gogrepo update" to update your GOG games, or "gogrepo download" to download only new games. You use both commands every time you want to download anything (new or updated) from GOG.
Thanks for the detailed reply, +1

1- ok for the bandwidth (but CTRL+C isn't certainly a solution).
2- uh, ok.
3- perfect, even if that "clean" command seems dangerous..
4- seems good.
5- simple downloads usually end up with tons of fragments, and I doubt Gogrepo is different without an option.
6- umm.. so, will manually renaming the folders cause re-download or not? (I need custom names, at least for them).

Also, does Gogrepo show a report about what it is going to do?
For example:
- new games: 4 for 70Gb
- update games: 2 for 20Gb
Post edited October 23, 2016 by phaolo
Posted October 23, 2016
avatar
phaolo: 3- perfect, even if that "clean" command seems dangerous..
The clean command will move everything it doesn't recognize (=find in the manifest file it created with the update command) from the target directory, and in its subdirectories, into a special directory !orphaned/. There you can check what it considered removing from your collection as obsolete files (and then delete them manually if you agree they are obsolete files). So yes it is a dangerous command if you'd run it into your C: root or the Windows directory (but then I am unsure what Windows would let you move from those places, unless you run gogrepo command prompt as an administrator).

However, you can first run the clean command with a -dryrun option. Then it tells you what files it would move away, without actually doing it.

avatar
phaolo: 5- simple downloads usually end up with tons of fragments, and I doubt Gogrepo is different without an option.
Yeah... but since most of your GOG installer collection will remain the same after the first download, you could just as well defragment them after that, and they should stay unfragmented afterwards? Maybe I don't consider this much of a problem also because my 2TB USB hard drive is specifically reserved for my GOG games, ie. it was initially empty when I started downloading my GOG games into it.

I am also unsure if there is any platform-independent way to do such preallocation, regardless of whether you are running the tool (script) in e.g. Linux or Windows, or any platform which can run python scripts? Gogrepo is not specifically a Windows/NTFS program, it is a python script that works on various platforms (which is one of the things why I like it, allowing me to run it both in Windows and Linux just the same). Some true python expert (like the gogrepo author) can probably tell for sure...

avatar
phaolo: 6- umm.. so, will manually renaming the folders cause re-download or not? (I need custom names, at least for them).
If you (or GOG for that matter) rename some game installer subdirectory or file name to something different that how it is in the gogrepo manifest file, then I presume gogrepo simply doesn't recognize that directory/file, and will redownload the said files all over again. Like renaming "age_of_wonders\" to "cage_of_wonders\" or whatever. Then gogrepo will be like "Huh? Cage of Wonders, what is that, some Nicolas Cage game? Oh, Age of Wonders is missing, maybe I should download it...".

Why do you need to rename subfolders or filenames anyway? There are some oddities in some of the directory names I admit, like some game had a "_copy3" at the end of the installer directory game name for some reason, while another game didn't have the game series name in the directory, maybe it was that Wallace & Gromit game...

avatar
phaolo: Also, does Gogrepo show a report about what it is going to do?
For example:
- new games: 4 for 70Gb
- update games: 2 for 20Gb
Generally yes. When you run the download command, it goes through your current GOG installer directory and lists of the files it finds in the manifest file, and tells for each whether it needs to be downloaded or not. I don't think there is any actual summary, but simply a long list of the files, both those which don't need to be downloaded, and those which will.

Also, as soon as it starts downloading, it will tell you all the time how many gigabytes are still to be downloaded, giving you the exact idea how much free hard drive space you need. The download command also supports the -dryrun option so you can get that list without actually starting downloading.

I recall the verify command will give you a summary of how many file integrity problems it found, and naturally the clean command lists the files it will move away as obsolete files.

Anyway I suggest you just try it out, it will all become clearer then. Also check the examples and options in the readme file (use Wordpad on Windows to read it, not Notepad), or in the github page.
Post edited October 23, 2016 by timppu
Posted October 23, 2016
avatar
phaolo: 1- ok for the bandwidth (but CTRL+C isn't certainly a solution).
1. I believe it is possible to throttle bandwidth using Powershell commands and so I may well add that to my script. I used gogrepo to download a new game yesterday and my wife complained that Netflix wasn't responding. I had to abort the download so it's definitely something I could use.

avatar
phaolo: Also, does Gogrepo show a report about what it is going to do?
For example:
- new games: 4 for 70Gb
- update games: 2 for 20Gb
The only report you will receive is what is displayed on-screen during and at the end of the gogrepo command being executed. The Windows Powershell script I have written saves the output from each gogrepo command to a text log file which allows me to the scan through the events at my leisure to (i) ensure that all went well, and (ii) allow me to view the games and files that have been changed/downloaded. Unfortunately there is no summary output as you show in your example but it can fairly easily be put together by reviewing the output from gogrepo which is quite detailed.
avatar
timppu: Thanks for the Windows script, I'll also take a look into it. I've made only a bash script for Linux for myself, but I was less familiar with Windows Powershell scripting, so I think I'll just change your script for my purposes. :)
Thanks - please let me know what you think.
avatar
ikrananka: The script will always "clean" the library but will only "verify" it if something changes in the library during the clean or download processes.
avatar
timppu: One thing I'd like for gogrepo is that it would keep track which files (or titles) it actually downloads, and then verify only those. It is just that full verification takes a very long time for me (usually the whole weekend for 1340 games), so I'd prefer normally verifying only recently downloaded games (or files).

For now I do it so that I track manually which games gogrepo downloads (either as new or updated games), and then I run verify for them one by one, by using the -id option.

EDIT: Now that I think of it, maybe it would be possible to do something like that myself in Linux. Grep from the gogrepo download command scan only those titles which it says will be downloaded, and then afterwards tell verify to check only those titles one by one, with the -id option...
That's a great idea. I too find that verify takes an awfully long time and would love to have it just do an incremental verify when I have only completed an incremental update. I'll have a look at modifying my script as well to see if I can get that working. I already have it scanning for the work "download" in the gogrepo download output, so I would just need to parse across to the game name from there.
Post edited October 23, 2016 by ikrananka
Posted October 23, 2016
avatar
timppu: So yes it is a dangerous command if you'd run it into your C: root or the Windows directory
Mm.. maybe some paths should be disallowed, just to be extra-safe?

avatar
timppu: you could just as well defragment them after that, and they should stay unfragmented afterwards?
The point was to avoid having to defragment 500+ GB of data.
Btw, Linux is probably more efficient about fragmentation, but the ol' Windows isn't.

avatar
timppu: I presume gogrepo simply doesn't recognize that directory/file
[..] Why do you need to rename subfolders or filenames anyway?
I imagined that too :|
Before Gog games, I used a script + naming conventions to dynamically list all my files and enable advanced searches.
Now, due to Galaxy, Gog's exe+bin combos and indirectly Gogrepo, I'm a bit fu**ed..

avatar
timppu: I don't think there is any actual summary, but simply a long list of the files
[..] the -dryrun option so you can get that list without actually starting downloading.
Mm.. a raw long list isn't ideal.
Does the "dryrun" command take a lot of time to complete?

avatar
timppu: Anyway I suggest you just try it out, it will all become clearer then. Also check the examples and options in the readme file (use Wordpad on Windows to read it, not Notepad), or in the github page.
Ok, thanks.
Maybe I'll also wait for some comment by woolymethodman and the guide update.

.
avatar
ikrananka: I believe it is possible to throttle bandwidth using Powershell commands and so I may well add that to my script.
[..] Unfortunately there is no summary output as you show in your example but it can fairly easily be put together by reviewing the output from gogrepo which is quite detailed.
Cool, thanks.
Post edited October 23, 2016 by phaolo
Posted October 23, 2016
Apologies, due to an overzealous use of Replace when editing my script, I broke the verify command. I have now corrected this and replaced the zip file I linked to here:

https://www.gog.com/forum/general/gogrepopy_python_script_for_regularly_backing_up_your_purchased_gog_collection_for_full_offline_e/post466.

If you did download my prior version then delete it and replace it with the new one which I have given a version number of 1.01.
Post edited October 23, 2016 by ikrananka
Posted October 24, 2016
avatar
ikrananka: https://www.gog.com/forum/general/gogrepopy_python_script_for_regularly_backing_up_your_purchased_gog_collection_for_full_offline_e/post466.

If you did download my prior version then delete it and replace it with the new one which I have given a version number of 1.01.
For anyone wanting to make a .BAT file to launch a .ps1 script you can use this command in dos to run it:
powershell -ExecutionPolicy ByPass -File X:\location\script.ps1
this will allow any .ps1 program to launch from any process as it calls powershell and bypasses default security.
Posted October 24, 2016
avatar
ikrananka: https://www.gog.com/forum/general/gogrepopy_python_script_for_regularly_backing_up_your_purchased_gog_collection_for_full_offline_e/post466.

If you did download my prior version then delete it and replace it with the new one which I have given a version number of 1.01.
avatar
Starkrun: For anyone wanting to make a .BAT file to launch a .ps1 script you can use this command in dos to run it:

powershell -ExecutionPolicy ByPass -File X:\location\script.ps1
avatar
Starkrun: this will allow any .ps1 program to launch from any process as it calls powershell and bypasses default security.
I'm using a desktop shortcut to run the script with "Target:" set to:

%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy Bypass -File "X:\location\script.ps1"

I guess this has the same effect as the batch file you're talking about.
Post edited October 24, 2016 by ikrananka
Posted October 24, 2016
@woolymethodman: What is the full list of language codes that gogrepo supports? They're not fully listed in the readme file or when running help on gogrepo.
Posted October 24, 2016
avatar
ikrananka: @woolymethodman: What is the full list of language codes that gogrepo supports? They're not fully listed in the readme file or when running help on gogrepo.
Open the python script in a text editor, it's near the top.
Posted October 24, 2016
avatar
ikrananka: @woolymethodman: What is the full list of language codes that gogrepo supports? They're not fully listed in the readme file or when running help on gogrepo.
avatar
woolymethodman: Open the python script in a text editor, it's near the top.
Perfect - thanks :)
Posted October 24, 2016
Having a problem with crashing during downloading. Using Linux Mint 18.

01:37:48 | 1053.59GB remaining
01:37:49 | 1591.1MB 0.3MB/s 1x lichdom_battlemage/setup_lichdom_battlemage_2.0.0.1-3.bin
01:37:49 | 1053.59GB remaining
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "gogrepo.py", line 881, in worker
assert out.tell() == end + 1
AssertionError

01:37:50 | --
01:37:50 | total time: 4:49:03.984103
01:37:50 | exiting...
Posted October 24, 2016
avatar
sbaylus: Having a problem with crashing during downloading. Using Linux Mint 18.

01:37:48 | 1053.59GB remaining
01:37:49 | 1591.1MB 0.3MB/s 1x lichdom_battlemage/setup_lichdom_battlemage_2.0.0.1-3.bin
01:37:49 | 1053.59GB remaining
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "gogrepo.py", line 881, in worker
assert out.tell() == end + 1
AssertionError

01:37:50 | --
01:37:50 | total time: 4:49:03.984103
01:37:50 | exiting...
Hmm sounds like it may have had issue writing the file out to disk. Do you have enough disk space? Is it always and only this file that fails?
Pages:
1
30
31
32
33
34
50
100
150
200
250
258
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3862 posts)(3862 posts)
(3862 posts)