If I abort the download (or it gets aborted due to internet problems etc.), and I later restart it, is there any reason to run verify or verify -delete before restarting the download?

I needed to restart the download (aborted it myself first) and decided to run verify -delete first so that it would delete the partial files that may be there, and also check for any other corrupted files (and delete them too). But it seemed to take so long time with this amount of files, so I aborted the verification and resumed with the download. At least it seemed to continue about at the same place where it aborted, showing around 720 GB remaining. I guess I'll run the verification afterwards.

What happens to those partial files that weren't finished, the download notices they are not ok, and just overwrites them? Or does it leave them be, thinking they are ok (until I will verify them myself)?

Ps. Overall the script is still running beautifully. The main feature requests I have are:

- The clean option like someone mentioned, e.g. that it lists and maybe moves orphan/obsolete files to another directory so I can check and decide if I want to keep them or delete them. Just in case GOG makes some stupid "update" like removing Outcast classic, or the DOS version of Earth 2140 from the extras...

- Proxy support. The internet at my workplace is like 6-7 times faster than my cable modem at home, sometimes that matters... :) I've been thinking of forking gogrepo and maybe trying if I can implement that myself to the current gogrepo.py, but I would be glad if the real python coding masters would do it instead, at least for the public version. I wouldn't probably want my amateur code to be included in the original. I could be a tester for that proxy support though, as I have means to test it obviously. (Well, actually starting this weekend, I will be abroad for four weeks, but before and after that.)

It just overwrites the partial files.

Once you start the download, it does a size check.
If the size is not the one recorded in the manifest, it will redownload it.

The Verify is for in deep md5 checking of the file contents.

The cleanup functionality is definitely needed. Just running a 90gb download, mostly what looks like a switch in how gog handles linux versions.
I already wrote a script to parse the gogrepo folder to get basic info into a sqlite db to process it further. So adding a clean functionality I can handle myself in a way, but that's not a solution for most using gogrepo.py.

I have no clue about python but realy hope that gets in soon.

I'm working on it... It's hard to find free hobby time for gogrepo between my full time job and life responsibilities. Too bad this isn't my full time job! :)

Anyway, my gogrepo changes in progress include:

- cmd_update() should detect if MD5 of new/unknown filename matches already known MD5, if so just rename the file rather than download a duplicate.
- cmd_update() should detect if game folder name is changed/renamed, but game id # is the same, then rename (could also verify that old folder name is no longer in new manifest data).
- cmd_clean(), new command to move unknown files/dirs found in gamedir to a orphan/ dir with same path structure (ex. the_witcher\old_patch.exe -> orphaned\the_witcher\old_patch.exe).
- cmd_clean() should allow skipping file/dir patterns

For ETA I hope by end of next week

I've noticed a few downloads that have problems (I think these might be on GOGs end but thought I'd mention them here):

The Defcon Executable always fail verification (but is the right size and passes its self-verification).
The Trine 2 Wallpaper pack is the wrong size (but downloads and opens, so it redownloads every time you do an update an download)
Fallout / Fallout 2 /Fallout Tactics have chunk areas since they were renamed to X Classic (I copied the contents from my previous folders and they all pass the size validation)
Eye of the Beholder 2 has a chunk area when trying to download it (seems to be some kind of discrepancy between the expected file size and actual file size).

For those cases I run an update on the game in question only, for example:
# gogrepo.py update -os windows -lang en -id trine_2_complete_story
And I got a new wallpaper archive, but on the next run it went through without update. They must have changed the file.
No errors with eye_of_the_beholder_ii_the_legend_of_darkmoon_copy3 and defcon for me.

OMG this seems quite a cool script!
I foolishly expected such features from Galaxy LOL.

I'll try it in the future, but I have some questions\requests for now:
- can it pre-allocate space to avoid bad fragmentation?
- can it auto-extract bonuses? I'm not going to keep everything and I'll store the interesting files elsewhere (eg:music)
- can it dowload files only if modified\added after a date (=update the collection)

Thanks and good job! : )

I decided to give this a try. Have previously used lgogdownloader for this very purpose, so to avoid downloading some 400G again, I setup an area with pretty much everything the way it expects already.

I find update is taking a very long time. I timed 52 minutes with 247 titles, when lgogdownloader gets the game info in just a couple of minutes. Is this because gogrepo uses the new API and is there room for improvement here?

Seems like an awesome script made in an awesome language!

Thanks a lot mate.

Can't wait to take a peak at the code and then start using it!

Guess we finally have a Linux backup client that is even better than the official one :P.

To download everything? Sure, I've used it for quite a while. I think this is the thread.

It's just that I think it's Linux-only and may be using old interfaces that are in danger of being deprecated for all I know.

Hi all,

To answer the recent questions in this thread:

"update" command takes a long time because I pre-fetch all MD5 info in addition to getting the game info data. I may add an option to skip md5 fetch for people who don't care about this. That would make update command be much faster, but you won't have MD5 data for doing a verify. I like to capture the MD5 data as part of the update command, because to me that's a "snapshot" of GOG database at the time you did the update.

For the trine 2 wall paper, that's a unique bug that I'll fix soon. Basically GOG put a new wallpaper file that has the same file name with a NEWER SMALLER SIZE. I need to ensure that trailing bytes from the older file are trimmed down. As a workaround, I suggest you just delete the file manually. It will be correct next fetch.

For anyone with defcon md5 errors, I emailed GOG about this a while back and they just fixed it (they had wrong MD5 listed in their database). Now it's correct, and the MD5 verification now passes! Kudos for GOG for fixing this in response to my support ticket.

The idea of pre-allocating space to minimize fragmentation seems like a good one. I'll probably add that.

I have work in progress and a few commits I haven't pushed yet, but still not much time :( Hope to have more soon!

Just replying as a satisfied user (EDIT: ok now I see the author replied too while I was writing):

I don't think it does, it just downloads as it goes (as I expect it to). It will tell you right at the start though how much data is (still) to be downloaded, but it doesn't do any extra check either whether the target drive has enough room (which I am not really expecting it to, I can easily check that myself as long as I know how much data is to be downloaded).

I don't think it does. Even if it did, it would still have to keep the original compressed extras around as well, so that it can verify which files are already downloaded. If it had only uncompressed files, then it couldn't verify them, unless it kept some extra database of the uncompressed files.

When you start the download, it will always perform a quick check at the target directory (using the manifest file you have), which files are already downloaded completely. It checks this by comparing the file sizes to the expected file sizes, it doesn't do any deeper md5 or zip file integrity check at that point.

If you download a whole new manifest file, then it performs the same check against that, and will notice if something is missing. Unfortunately it considers also a mere game directory name change as a new set of files, like when a couple of weeks ago GOG changed "the_witcher_3_wild_hunt" download directory name to "the_witcher_3_wild_hunt_game", the downloader was about to download all the TW3 files all over again. I noticed that though, so I simply renamed the TW3 download directory at the target, after which it considered them ok and didn't try to download any existing files.

I wonder why GOG made that TW3 name change? Is there possibly going to be also a The Witcher 3: The Movie?!? You heard it here first, folks! :)

Since downloading a whole new manifest file can take a lot of time (with e.g. 1023 games, it takes me the whole night), there are these options to update the manifest file partly (from the readme file):

* Add new games from your library to the manifest.
``gogrepo.py update -os windows -lang en de -skipknown``

* Update games with the updated tag in your libary.
``gogrepo.py update -os windows -lang en de -updateonly``

* Update a single game in your manifest.
``gogrepo.py update -os windows -lang en de -id trine_2_complete_story``

However, if there are some old files that have become obsolete (either the file name has changed, or the file is not used anymore), then those are currently left hanging around. A clean option to move or remove such unrecognized files (by the manifest file) is requested, but currently the only way to get a clean set that get rids of the orphaned files seems to be to run "gogrepo,py backup" (or "gogrepo.py import", actually I am now a bit unsure what's the difference between those two commands...)

``gogrepo.py import`` Search an already existing GOG collection for game item/files, and import them to your
new GOG folder with clean game directory names and file names as GOG has them named on their servers.

import [-h] src_dir dest_dir
src_dir source directory to import games from
dest_dir directory to copy and name imported files to
-h, --help show this help message and exit
--
``gogrepo.py backup`` Make copies of all known files in manifest file from a source directory to a backup destination directory. Useful for cleaning out older files from your GOG collection.

backup [-h] src_dir dest_dir
src_dir source directory containing gog items
dest_dir destination directory to backup files to
-h, --help show this help message and exit

Maybe import is deeper (and slower), ie. it doesn't expect file and directory names to be correct in the source. Otherwise the end result is the same, I presume.

I hope a clean option will come, as my games currently take something close to 1.3 terabytes, meaning I temporarily need twice that space in order to perform either import or backup.

Fully agreed. This has turned out to be a very useful utility for someone like me who likes to keep a local backup of his DRM-free games. I was going to check lgogdownloader, but gogrepo is easier if you want to work also with Windows, I guess. I don't know if there is any Windows user for lgogdownloader, but I've used gogrepo.py in both Linux (Mint) and Windows (7). In Windows it took a bit of research to find out what are all the dependencies that need to be downloaded and how, in Linux Mint I think I needed to download/install only html5lib from the software source, and mark gogrepo.py as an executable (chmod +x gogrepo.py), then I was all set I think.

Thanks for the reply.
About the folders name change, can't gogrepo maybe use game IDs like Magog?
Also, my suggestion for update>date seems useful if you just want to get new\modified games without checking old data. (Eg: download 15 new titles, but ignore your old 800)

There are some problems with only getting new games / updates:

GOG doesn't mark things as updated in many cases where it should (eg Witcher 3 DLC doesn't , and changes to extras also don't). Which is a little (okay a lot) annoying since it means you have to do a full update to be sure everything is correct.