I am working on yet another unofficial gog client. It can be found here: https://github.com/Magnitus-/gogcli

If you are already satisfied with another tool for your backup needs, I encourage you to keep using it (a backup tool is a backup tool).

If you have yet to make a decision or found other tools to be unsuitable for your needs, here are highlights of this one:
- It supports two different storages (more could be added): Filesystem and s3 protocol
- It is user friendly to install (pre-built binaries that can run standalone without any runtime dependencies)
- In addition to providing the usual set of commands to manage your backups, it also provides lower-level commands to interact with the gog api, which might be interesting for scripters

The main drawback the tool has at this point is that it doesn't have a user-friendly mechanism to authenticate. It relies on you copy-pasting values from your browser cookie once you are logged into gog. I plan to eventually address this issue, but it will not be immediate.

Cheers.

Related Projects:

A Windows GUI built on top of gogcli to manage your games with a user interface, implemented by Timboli:

https://github.com/Twombs/GOGcli-GUI

Keep in mind that the above tool by Timboli makes use of gogcli mostly to interact with the GOG.com api and to generate the initial manifest. After that, it goes on its separate way implementation-wise, so there is a significant difference in how the backup files are ultimately managed compared to the "storage" commands of this tool.

Release:

v0.2.0:
- Fixed bugs that occurred when the manifest has more than one game
- Added an optional argument to limit the number of games that are uploaded in a storage or copied from one storage to another (the former allows to upload your games across several iterations instead of all at once, the later allowed me to validate the functionality in a timely manner without constantly pinging the gog api)
- Added a "gogapi storage resume" command to resume storage upload or copy which is pretty much a pre-requisite to go along with the above argument

v0.3.0:
- Added an api command that retrieves checksum and size information on relative download url in addition to the file name (when possible)
- Added checksum and file size inclusion in generated manifest for the files for which gog provides the info
- In addition to the separate "storage validate" command, checksums and file sizes will also be validated against the manifest during file uploads
- Fixed an edge-case bug that would occur if an upload fails and you resume it

v0.4.0:
- All files in a generated manifest will have a verified size info derived from the GOG api and verified size will be computed at the game and manifest level
- When a checksum is not present (which will be the case when doing an initial upload from GOG, but not when copying between storages), zip archives will be verified for correctness

v0.5.0:
- Added "update generate" command to generate an update file (a list of game ids for new games and updated games as indicated by the gog api)
- Added optional json and file output for most gog-api commands

v0.5.1:
- Fixed a problem that would occur when generating large manifest due do http handles not getting properly closed
- Added resiliency against malformed xml coming from gog when getting files metadata

v0.6.0:
- Better title filtering for manifest generation
- Added manifest generation filter to generated manifest
- Added fault tolerance during manifest generation for dangling download files that are listed by the gog api, but no longer downloadable

v0.7.0:
- Fixed observed lingering issues with manifest generation
- Fixed potential problem with long downloads with download handles not closing
- Added command to get a summary of a manifest (useful when your manifest has 1000+ games)
- Added commands to retrieve the manifest and actions files from a storage (mostly a convenience thing)

v0.8.0:
- Fixed a lingering bug where dangling files (ie, empty files that gog needs to cleanup) would still be included in the manifest as incomplete file
- Added a directory argument for the "gog-api download-url-path" command
- Added some unit tests (about ~30% coverage in the manifest package)
- Added a "manifest diff" command (similar to "storage plan", but can be run between two manifest files without having a storage)
- Added optional argument to many commands to tolerate empty checksums if other file parameters like name, size and url match (would previously only support the paranoid approach of assuming the files are different if checksum is empty)
- Added a "manifest update" command to update a manifest from the gog api with an update file
- Added a "manifest search" command to search a manifest with the same search parameters used for manifest creation supported.

v0.9.0:
- Added version command to get the binary's version
- Fixed bug that would cause game file delete not to appear in the debug logs for the filesystem storage
- Fixed a bug that would cause checksums not to be properly computed for files larger than 64MB for the s3 storage
- More unit tests (over 50% coverage for the manifest package)
- Added a "storage update-actions" to update ongoing manifest upload with manifest updates from gog.com
- Added flags to order game downloads for the "storage apply" and "storage resume" commands
- Made game file deletion for all storages more robust by tolerating the game file being absent without terminating in error

v0.10.0:
- Added support for duplicate filenames using a "[]" delimiter
- Added optimisation to compact the same installer listed multiple times in multiple languages into a single entry (with all languages as an array)
- Improved logging structure to support several different log levels (although for now, only debug is really used)
- Added manifest migration command to migrate manifests generated with gogcli version 0.9.0 to the latest format
- Added storage repair command to fix any accidental divergences between the game files and the manifest

v0.11.0:
- Added retrying for most sdk calls (but not downloads)

v0.12.0:
- Added architecture documentation to better understand what the main commands do
- Added support for netscape-style cookie files

v0.13.0:
- Added support for retries when downloading files
- Improved logging: Debug logs are nicer and added what I felt is the right amount of info logs when downloading files

v0.14.0:
- Added download speed measurement for the standalone "gog-api download-url-path" command

v0.15.0:
- Reworked the storage commands in a way I felt was more intuitive (fewer commands). Also fully separated the application of a manifest to the storage from the execution of the underlying actions, allowing users to inspect what will be done before it gets done and opening the door for third-party tools and scripts to do custom modifications

v0.16.0:
- Added command to get a summary on an actions file
- Added a performance indicator to the standalone api command to download a game file
- Made manifest applies safer by default (no game deletion allowed unless specified via a command flag)

v0.17.0:
- Changed storage verification and repair commands to skip md5 checksum verification by default and added a command flag to enable it
- Changed the repair command to support using the manifest directly in the storage and made that the default case as it is the more common use-case.

v0.18.0:
- Added a fix for files with bad xml metadata with an optional flag

v0.19.0:
- Added game slug to manifest files and a migration path to add it to pre-existing manifests
- Made the flag to compensate for bad xml metadata the default

v0.20.0:
- Added progress logs for manifest generation/updates
- Added commands to resume manifest generation/updates
- Added more context information in generated (for manifest generation/update) warning files to indicate the action gogcli took (metadata workaround or skipping the file)
- Added bad metadata workaround to gog-api url-path-info command
- Fixed bug for the storage plan command (must have been broken for several releases)
- Fixed reported Windows newline format bug when reading user-generated cookie file
- Fixed quirk where errors would be writen in the manifest file when manifest generation/update halted with an error

For a list of my more immediate TODOs, see the issues: https://github.com/Magnitus-/gogcli/issues

Legacy TODO list (I'll stilll get around to a lot of the TODOs below, but not sure when):

Planned major improvements:
- Provide separate metadata commands to download games metadata (images, slugs, ideally video links and description as well)
- Provide a web dashboard to give an experience similar to your gog library page when viewing your backup
- Provide a nice login solution without compromising the ideology of the tool (self-contained binary, multi-platform, format that is friendly to manual tweaks for DIYers when certain mechanisms fail)
- Provide a "--id" flag to the "storage repair" command to save time and only repair individual games when it is known that only specific games are inconsistent

Planned minor improvements:
- Change gog-api owned-games command to retrieve all pages at once
- Add a custom duplicate delimiter parameter to be able to specify delimiters other than "[]"
- Add a "--closest-title" flag to the "manifest generate" and "manifest search" commands to return only a single game whose title is the closest match to the specified search term

Uncertain Improvements:
- Support other downloader storage formats via additional storage implementations
- Provide a GUI for most command line commands via a web dashboard

Certainly saving this for future use. Thank you!

My pleasure. I try not to be overly optimistic with time estimates, but interpolating from the fact that I've been working on it for about 1.5 months, I should have something reasonably stable within the next 2-3 weeks.

I'll post release updates in the thread.

Good idea to relocate, as we certainly ended up hijacking that thread ... for good reasons of course, as this stuff is all related and several gogrepo.py users have good insight, as Geralt_of_Rivia, Kalanyr and others have demonstrated.

Hopefully they will follow you here from -
https://www.gog.com/forum/general/gogrepopy_python_script_for_regularly_backing_up_your_purchased_gog_collection_for_full_offline_e/post2693

Certainly for any GUI frontend that I would create in Windows, I would add support for some desirable missing features, while trying to keep them as simple and painless as possible.

Geralt_of_Rivia quoted. That seems promising maybe for doing multi-thread downloading per file.

If you use Autolt, you're joined to Windows at the hip I think. Not much you can do about it and no point beating yourself up over it. I locked myself up with Windows when I learned C# around the time it came out too.

On the upside, you can support windows-only thing :P. Given that home internet bandwidth tends to become saturated very quickly (even if your internet connection is out of this world, I believe gog will throttle your download speed to 3mb/second so downloading two files at once from gog only means they'll both be downloading half as slow), I don't think much would be gained from that optimisation unfortunately.

The client does support downloading multiple files in parallel, but given the throttle, even that might be a pointless optimisation.

I've typically not been throttled. Anecdotal for sure, but my overall speeds are right at 100 Mb/s (11-13MB/s)

I regularly reach speeds of 250Mb/s, equally divided between the files I download (up to four in parallel, usually).

Actually, you're right, its probably my home internet speed. I keep forgetting that its in megabits per second and not megabytes per second which would actually put it at ~3.1MB/s.

On a side note, if you have 250 megabytes per second, either you download at work, the internet where you live is light years ahead of the internet where I live or I just don't want to see your internet bill :P.

Still, as impressive as it is, even at that speed, if you download a significant part of your collection, you will still saturate your internet bandwidth pretty quickly.

Not 250 megabytes (MB) per second, 250 megabits (Mb) per second.

Right. Never paid attention to the capital vs lower case distinction (guess that speaks about how often a byte base unit is simply assumed that you don't need to question it), but its true, in networking bandwidth, its all in bits as the base unit.

That does put your internet speed at a more human level.

Edit: Fascinating, that does explain while some large file transfers on my home network was not quicker. I needed to divide by 8. Its not 1GB/s, its 125MB/s ><.

Edit 2: Actually, even slower than 125MB/s, since that is the capacity of my nics and switch and the cat 6e wires are even slower than that. I vaguely recall that you can treat 2 physical ports as one doubling your throughput if both your nic and your switch support it... the name is not coming back to me now, though maybe I need it more than I originally thought.

Edit 3: Just Googled it. Its link aggregation. I guess I could benefit from bushing up on my networking more than I thought.

Not sure if you know, but gogrepo.py is set by default to download 4 files at a time, which for my preferred scenario, I have limited to just 1 file at a time.

See this line (140 for me) in gogrepo.py - HTTP_GAME_DOWNLOADER_THREADS = 4

In discussions a while back with Kalanyr, in regard to adding an option to do more than one thread per file, he declared it would be complex to set up, and not something he felt inclined to do.

I cannot say how complex that would be as it is right out of my comfort zone, any network stuff.

But going by the chunks we have now seen listed care of Geralt_of_Rivia, I imagine that anything that might be able to download a game file in chunks, and then combine the result after checking each MD5, could mean it is not too complex ... I am of course speaking from inexperience and it is just conjecture on my part. I'd have no idea how to download chunks and not entirely sure how to properly join them either ... though from some vague recollections of ffmpeg for instance (no help to us), I imagine it would do the lot for a video file ... if you knew how.

I default to 10 and it is configurable directly from the command line: https://github.com/Magnitus-/gogcli/blob/main/cmd/cmd-storage-apply.go#L60

For splitting the file into several downloads, it is complex, but doable (but it would slow down checksum computation as currently, I use the same download stream to both download the file and compute the checksum, but I could not do that it was downloaded in parallel parts as I believe each iteration of the checksum computation is dependant on the previous iterations).

However, I'm not sure there overall gain would be appreciable. If you are downloading a bunch of files, you already have parallelism across several files.

Furthermore, none of this probably matters though, because what you might speed up when you create multiple "threads" is downloading a bunch of smaller files (smaller than your bandwidth capacity) as there, the http connections and processing overhead is possibly (as in, you'd have to do benchmarks to verify this) significant compared to the size of the download and you can parallelise this.

On a whooping download (ex: a 8GB installer), even a single "thread" will saturate your internet connection, downloading at whatever your internet speed is. If you add more "threads", your will just make each download slower, but won't increase your throughput. Not only that, you might make actually make the whole thing slower (though probably not appreciably), because you'll incur several needless http connections overhead on the same file.

Now, if you have a server, it is more interesting to make it async, because the server could serve smaller requests for other clients while its doing a big download for one client and not let the connection queue pile up or give a bad experience to all those clients making a smaller request, but the point is moot here, because your backup client is dedicated to you, it won't be doing small things for other people on the side.

As it is, even across files, I might just be supplying the user with false illusions that they are getting a lot of throughput out of their multiple concurrency executors, but in reality, they might only experience a small speedup if they download several smaller files as their internet bandwidth is the bottleneck. If I broke a single file into multiple downloads, I would just be straight out lying to them instead of giving them a half-truth. It won't be faster.

Ok, if I find out there is a way to get a full installer, I'll investigate it, but if its just that 1mb "pointer" file, then I won't bother.

Totally agree, and I am only interested in a full size Galaxy variant if the worst happens ... GOG get rid of offline installers. I guess we cross that bridge if it happens, but your program and gogrepo.py variants become redundant instantly if GOG do that, and no other offline option exists.

Many of us would tolerate Galaxy features in a full size installer file, but not tolerate a lack of a local offline installer file to backup.

I cannot see GOG doing that, but don't believe in having all my eggs in one basket, just in case. Certainly GOG would be stretching the meaning of DRM-Free, if the only way to do a backup is to backup the entire game folder ... which would be a right pain in the ass, and no good for updates ... which would require keeping the game installed or a fresh install of it, to then freshly backup.

What you say makes a lot of sense I guess, and I am not a fan of even doing multiple files at the same time. I like to get each file as soon as I can, especially with the variations in speed and reliance since around the time COVID happened.

Never been sure if that is just due to GOG or whether my ISP is also to blame ... I suspect both to some degree. This is especially so, when I do my downloading in the early hours here ... that should mean no issue my end, with very few online here at that time. GOG do seem to bias servers for certain countries at specific times. Big sales only make that more evident.

To be perfectly honest, if I am downloading a large game file or many of them, I just queue up in Free Download Manager 5, which has throttling, multiple threads and resume, etc. I do tend to grab all the small files and game cover with gogrepo.py though, and create the game folder (though I later rename that), and also use it to do the verifying.