Version 0.10.0 is out. See first post for details.

Also broke down what the gameplan is in terms of expected future improvements for the tool.

Fun fact: I have 985 out of 2084 games left to download, meaning I'm past the median. I ordered them by size (meaning smaller games get downloaded first) and I only downloaded 599.86 GB out of ~6.5 TB of games so far (meaning less than 10% of the total size).

There is obviously a very large size disparity between indie/retro games and modern AAA games.

Indeed.

I am about to download the final game (Mad Max) of my recent GOG acquisitions, which included 13 demos and 6 purchased games. Saved the largest until last - almost 32 Gb.

This will be the first large game I will have tried with gogcli.exe.

Usually these days, I download such games with Free Download Manager 5, and only did the smaller game files with gogrepo.py. But thought I would see how we go with gogcli.exe ... especially during a big sale ... I got varying download speeds all over the shop yesterday with the other games and gogcli.exe. Sales can be a real slowdown I find and very dependent on time of day in your region.

As usual, I've left it until late in the day here (after midnight). So if too slow I will cancel and resume tomorrow ... hopefully at a better time.

One thing I noticed is that downloads are more reliable if you reduce parallelisation. I think if the download takes too long, the gog server gives up on it at some point and terminates the connection so depending on the transfer rate, its probably better to have fewer concurrent downloads.

These days, I set concurrency to 2 before bedtime and at that concurrency level, I've yet had it terminate due to gog prematurely closing the connection. However, I think the largest file I downloaded so far might have been ~2GB.

I only download one file at a time using my GUI and gogcli.exe.

I managed to get 3 x 4 Gb files before I went to bed, and speed seemed okay ... some issue with my code aside when reporting that.

Today I did the other 5 x 4 Gb files, and after the first two, delved into my code and discovered a bug (forgot to reset the timer initiate after exiting the size determinate loop). So first download reported correctly, remaining ones didn't. All fixed now and some additional improvements to the progress bar.

The remaining 3 x 4 Gb files downloaded reporting an average speed of 5.2 Mb/s. Which is close to the maximum I ever get.

Can gogcli autogenerate a directory tree like lgogdownloader does?

ie I want to have that all stuff from a game should have its own game directory with the slug name? Or does gogcli just dumps everything in a single directory?

It generates a directory structure. However, it uses the game id, not the slug.

It looks like this:

games/
--manifest.json
--[actions.json]
--[source.json]
--<id of game 1>/
----installers/
------<installer file 1>
------<installer file 2>
------...
----extras/
------<extra 1>
------<extra 2>
------...
...

If wasn't sure if slugs were unique per game (I try not to assume things), but if they are, I could potentially integrate them in the manifest and use them for the directory names which would make it nicer to the eye.

Either way, my ultimate plan is to strap a web dashboard on top of everything to download games from your LAN which should make it user-friendly enough to traverse.

slugs are unique to each game.

The only problem with GOG is that they originally had no concept of DLC in their structures and they were just another appid linked to the game appid, but I'm convinced that you already know of the painful mess ;)

For me, it is more important to have an accurate offline mirror of my games, neatly packaged.

I know that it is a bit much to ask, but it would also be a great boon to have the directory structure itself configured, like lgogdownloader does? My preference is now that I just dump everything into the slug directory, for easier handling (had it sorted out in the beginning, but this always duplicated files unnecessary. there were always soundtracks in extras, some in patches; language packs some in languages, some in patches, some in the game directory, so I decided to ditch the substructure for saving space -- at least with lgogdownloader).

It would be great to offer some flexibility in this regard.

Certainly the slug was what the good old GOG Downloader used for download folder names.

My GUIs for gogrepo.py or goccli.exe certainly support using either the Slug or Title as the game folder name. They also support using an additional Alphanumeric sub-folder name to store under.

I use the Slug for my Linux downloads and the Title for my Windows downloads, from GOG.

I have considered doing an Extras sub-folder and Patch sub-folder, but not come to a decision yet. I certainly often create them manually for some games that have a lot.

It won't be immediate, but I can look into it. Atm, the storage implementation has an interface and the storage internals are mostly decoupled from the rest of the app.

However, the interface specifies the game id as an argument so this might require some refactoring: https://github.com/Magnitus-/gogcli/blob/main/storage/storage-interface.go

I guess something that I could do is implement a "CustomFs" storage that keeps a mapping of ids, slugs and other stuff and allows you to customize your paths and then I wouldn't have to change the interface.

The other part is that currently, the manifest doesn't contain slugs at all. I toyed with the notion, but because of the way the GOG api is structured, opted to not have it for now in order to shave up some seconds during manifest updates. So, I'd have to refactor that as well.

Otherwise, in terms of the underlying model that permeates the client, the "game -> installer | extra -> file" structure is more important and somewhat less flexible: https://github.com/Magnitus-/gogcli/blob/main/manifest/manifest-game.go#L17

In particular, file duplicates are detected and resolved by clashing file names within a given "game -> installer" and "game -> extra" category.

If you eliminate the installer/extra directory structure in the storage, you might end up with duplicate file names (in the unlikely event that for a given game, an installer and an extra have the same file name) that are not handled in the model.

Well, I have some sort of scraper for the GOG shop written in Python which parses the GOG manifests and they do have the slug in them. You also don't have to own these games in order to parse the data, only if you wanna have the download meta data (which is in this case not a concern):

For example, I'm using the game "Ultimate General: Civil War" (id: 2129139688):
-- snip --
"slug": "ultimate_general_civil_war",
"supportUrl": "/support/ultimate_general_civil_war",
"supportedOperatingSystems": [
"windows",
"mac"
],
"title": "Ultimate General: Civil War",
"type": 1,
-- snip --

You can parse the current store data (first page, have to iterate until you have it all; and you should do some caching on the client side) at https://www.gog.com/games/ajax/filtered?mediaType=game&amp;page=1

In addition, there is no overlap in file names concerning extras / installers :)

Anyway, it would be real neat but I can understand the concerns on your side.

Though your owned-games command does return the Title and Slug. Something coffeecup may not realize, and which my GUI uses.

I looked at what the gog api returns with curl (when in doubt, curl almost never lies) and I can confirm that the unicode replacement for some characters (notably &) doesn't come from my client, but directly from the gog api.

I'll fix it for a given set of characters by adding some substitution to manifest finalization. I suppose if I want to be gun-oh about it, I might just do a regex for all unicode characters and replace those that fall in the ascii realm (I guess its not too common that you'll see \uxxxx in a title and it meaning something other than unicode substitution). I'll see.

Its very unlikely, but given that certain games have installers in the extra and that file metadata appears to be managed by hand, duplicates would be possible.

So far, I encountered:
- Malformed xml when fetching file info
- Dangling files in the library that give a 404 or 403
- Duplicate filenames (for different files, not the same file under multiple languages)

Nothing surprises me anymore.

I can't tell you with a straight face that I'll do it right away, but I think its a reasonable request.

I can't give you a firm timeline (its a nice feature, though not what I would consider a core one at least not if you view the backup more like a database and less like a gui that you will manually fiddle in... in my case, I'm doing my backup in an s3 store and then will implement a dashboard to access my games so this is very much my approach), but I'll find time for it.

Oh yes. I'm just not using it in the manifest right now. His request is definitely not outlandish to implement.

Otherwise, given his reference to lgogdownloader, I strongly suspect that like me, he's on Linux.

Goodo.

Yep, and I wasn't suggesting he use my GUI, as lgogdownloader is indeed a Linux giveaway.
More it was just showing that there was a way to get the Slug, in case that wasn't known.
And also from my point of view, you need Titles to get manifest entries anyway, and while getting titles, you by default get the Slugs as well. How you (as a user) tie that into use with your manifest, seems to be the only question, but you are doing so with titles anyway, so shouldn't be too hard.

Having slugs in the manifest entry as well, is a great idea though. :)

P.S. Talking about manifest entries, I have now added over 600 of my 1200 to my manifest, plus weeded out all the earlier problem entries and replaced what needed replacing.

I noticed something in my manifest yesterday, and just mentally took note of it, because I was busy checking something else.

Today I went right through my manifest, looking for other instances of what I saw yesterday, and found a few more, but only minor versions of it.

Basically, it is a space or more at the end of a title in the "Title": field (row / line).

The most extreme example I came across yesterday, and no doubt why I noticed it, was for the Fantasy General II: Prologue game. Note the following.

"Title": "Fantasy General II: Prologue         ",
"Name": "setup_fantasy_general_ii_prologue__________1.02.11096_(40120).exe",

The following games just had a single space after.

"Title": "Bombshell ",
"Title": "Bombshell (Part 1 of 3)",
"Name": "setup_bombshell__1.2.10466(ion_furry)_(34762).exe",

"Title": "Crusader Kings Complete ",

"Title": "Jazzpunk: Director's Cut ",
"Name": "dlc_jazzpunk__flavour_nexus_en_Build_9_16121.pkg",

I also noticed for the first time, that "CdKey": does actually have something in it sometimes. It never occurred to me it was Serial or Redeem Code related. I should now add something to my GUI to record and display it.

I am guessing this is a fault GOG's end?

P.S. Damn forum software screwed the spaces. So had to use underscores in the Fantasy General "Title": field to represent spaces. Those underscores in the "Name": field do exist though (no spaces there). EDIT - Actually 'nbsp' worked for the 9 spaces.