No worries.
I'll cross my fingers and toes and hope for the best. :)

I've just come across a scenario I've not come across before - the same game with two different IDs.

https://api.gog.com/v2/games/1272776844

https://api.gog.com/v2/games/2060132780

I came across that when I noticed my cleanup routine for wishlist entries in a program of mine, wasn't removing that recent game I bought.

It's the not "same game" from GOG's POV.

One, ending in 780, is a game. It's commonly referred to as "the library ID".
The other, ending in 844, is a package that contains the game (and at some point possibly contained something in addition to the game, e.g. a pre-order exclusive). It's commonly referred to as "the store ID".

There are tens of such pairs on GOG, especially common for games with pre-order or kickstarter exclusives.

Another side effect of this, beyond the orphaned wishlist entry, is that such games never get an update flag, because of the double ID, so you need to either follow the "what did just update" thread, or check manually from time to time.

Wow, thanks for that bit of info, I was not aware of it.

I guess from time to time, I'll do a factory update, regenerating the entire manifest then.

Ok, thanks for the info.

Of course, for my wishlist program, it gets the ID straight from the game's store page. Not sure if that ID itself ever changed over the course of time, or whether the manifest one was always different. Though what you are saying, implies it did, and maybe initially they matched each other.

Initially for my wishlist program, I wasn't grabbing the web page ID, and have only been doing that a few months now, and retrieved the missing ones back when I implemented that facility. That game was one of the missing ones for sure.

Unless your program does something different to what it does now when retrieving detail, I don't think any change is going to be detected.

Every time I do an update with my GUI using your program, all the Titles and IDs are obtained and populate my lists. That process has a check, and if the Title plus ID combo isn't found, a new entry gets created.

I have no such new or alternate entry for that game.
So the only ID entry for 'The Uncertain: Light At The End' is 2060132780
And that doesn't match the current store page ID of 1272776844, which can be seen by looking at the current source of that page, where 2060132780 is nowhere in existence.

https://www.gog.com/game/uncertain_light_at_the_end

So I don't know how this is going to get rectified, without maybe a duplicates check of some kind ... presuming there is another manifest ID available. Perhaps the two are just out-of-sync and that's it ... in sync initially.

Not that it needs to get rectified as far as getting manifest detail and download links, because I did that fine a few days ago.

It would appear, that only the store page ID has changed, and the manifest ID has remained the same.

You are mostly using the client for api calls, so you may diverge here.

However, if you use the storage commands, there are two main paths you can take for updates:

1. Trust GOG To Report Everything

Basically, you generate an update file based on what GOG reports as New or Updated.

From there, you download your manifest from your storage, update your manifest for those specific game entries and then apply your updated manifest to your storage. The later will generate a 'action' file which are actions to take based on a 'diff' between manifests and execute each of those action on your storage.

2. Just Update the World

Basically, you generate a new manifest from scratch.

Then, you apply the new manifest to your storage with one important caviat: You use the flag that tells the client that if a checksum is missing from the new manifest (which will be the case for most extras), it should trust the file to be the same assuming that all other characteristics (including name and size) match. For zip files, a check on the integrity will still be performed. If you don't do that, most of the extras will be redownloaded and if your collection is large, you don't want that.

In the above case, it doesn't matter what GOG tells you is updated. It will detect all changes and apply them. If they went so far as to change the identifier of a game, the client will just interpret it as an addition and a deletion instead of an update, but the end result will be the same, with a little more work having been done.

I usually go for option #1 myself, but a handful of time, I have gone with option #2.

No worries, thanks for the info.

But I was kind of saying, that update issue aside, the only impact was probably my scenario where I am comparing store ID with manifest ID. So if just using your program, I imagine there is no issue, aside from the update possibility.

My GUI can be set to get the latest manifest entry before downloading, and what shows up in the downloading window is what isn't recorded in the database my GUI also creates with every download. So in essence only new files are shown. On top of that, my GUI also checks if the file downloading matches the file name in the current manifest entry, and alerts if that is not the case. It may even re-download the whole manifest entry for that game ... I don't recall ... but pretty sure the manifest is at least updated for each file instance.

P.S. Still, it would be interesting to see if your program comes up with a manifest entry for the store ID, and whether you do end up with two of them for the same game.

It's not the only type of game that never gets flags.

For example, unbundled games (i.e. where the store sells one item, while the library gets several boxes, e.g. Ultima 4+5+6, Tomb Raider 1+2+3) do to, but they rarely get updated.

And there's the occasional game where GOG changed the library or store ID after some time for reasons unknown, but these have been rarer in recent years.

And finally, changes to bonuses/extras/goodies (added bonuses, updated bonuses) also never get flagged in any game.

Might be a bit late in the game at this point, but assuming you don't mind doing some editing on the manifest with your tool to fill in the missing checksums in the extras (currently, those only gets filled when you upload extra files to the storage) and potentially some other stuff you want to customize (ex: trimming extras, installers, etc, etc), you could potentially leverage the "manifest diff" functionality of gogcli, which, imho, is pretty nice (it will generate an action file which can be further edited to your needs or even shown to the end users for them to review what will happen).

For the "standalone" version, it is the "gogcli manifest diff" command. It doesn't require to use the "storage" functionality of the tool, just 2 manifests to compare (it will output an actions file).

Both the "manifest" and "actions" file are in json format and easy to manipulate so you could potentially add a lot of custom fonctionality on top of what the tool provides.

I will check when I get back.


Thanks for the info. I can only bow to your knowledge of gog update processes.

Concerning your last bit of info, I did notice that when you buy things like a soundtrack dlc (and I believe other dlcs as well), that never gets shown as an updated/new in your library. Its pretty much the reason why I just update from a new manifest from time to time. I get worried that I missed one of those.

No worries, I'll have a bit of a think about it and maybe investigate.
Thanks for the info.

It is disconcerting to know, that I may have missed out on things I have not been alerted to, and I have discovered a few over time (i.e. updates to The Long Dark etc).

However, when I buy a DLC, I always download it immediately like any game, and because I know it has been added to an existing game entry, I do an automatic manifest replacement to pick up the new download(s).

So just a bummer for me about any unknown extras. And because of the way I store my games, not an easy thing to code a solution.

GOG don't make it easy sometimes. Especially when you buy something like a Special Version (etc) of a game, but the game is only listed as the vanilla version in your library. Generally, If I am paying attention with my downloads, I manually rename the game folder to match the Special Version (etc) name. That name is also what goes in my Excel spreadsheet, with a breakdown of game parts, so the Vanilla name also gets listed.

I've downloaded 160 games with 44 games to go (lots of updates while I was away).

The download speed seems to be acceptable, though I haven't taken exact measurements yet.

Something that I've noticed is that the GOG server seems to hang up a lot (transient download size and checksum mismatch, every 10-30 games downloaded), enough so that I'm now working on automatic retries for downloads. I'll take the opportunity to improve the info level logging while I'm at it (the debug level logs are serviceable, though a tad chatty).

The above will not impact that part of the client that you are using however.

Once I'm done with the above, I'll see if I can do something to add a speed measurement to the file download standalone command to measure download speed more accurately.

PS: Otherwise, I've added support for Netscape format cookies (may or may not be more user-friendly for you, I know TheDcoder wanted it and I believe someone in the past requested it as well and it wasn't a lot of work to implement so...).

Also added some high level architecture doc for more clarity, though technically it mostly focus on the storage.

No worries, thanks, I will check it out.

I'm still having download speed issues. Sometimes it is like yin and yang, with one download being an okay speed but the next quite slow, then maybe another faster download, then slow again etc. Though I get more slow downloads than okay speed ones, and it seems more often than not, that the BIN files download the slowest, which is painful with big files. It used to be that game extras downloaded the slowest, but for some time now I am finding they are usually the fastest.

Generally the download speed settles to a certain value and stays around that ... whether that's a good speed or slow speed, and I've not had any failures. I have tried stopping and restarting to overcome the slow download speeds, but I rarely if ever get a benefit from that.

The improved logging functionality (you won't benefit from the info logs when running the storage actions, but you should try the debug log level for the api commands, I think you'll find it a bit better to look at than before) and download retries are working quite well. I'm pleased. The tool is starting to approach something I'm satisfied with.

I cogitated about the various things related to the tool yesterday.

Storage Commands Refactoring:

That one is more for me.

I think the commands for the storage are more complicated than they need be. I think I should streamline the "storage apply" + "storage update-actions" + "storage resume" commands as follows:

storage apply manifest: It should either create an actions file or update it in the storage depending from a fresh manifest, nothing more (basically, part of what "storage apply" is doing and what "storage update-actions" is doing)
storage execute actions: It should run actions on the storage (basically, part of what "storage apply" is doing and what "storage resume" is doing)

Download File Api Command (to measure and improve download speed):

This, you'll be more interested in as it pertains to your issue.

I thought about providing a continuously updated download speed indication in real time as the file downloads to measure the speed (ie, something approaching a derivative), but this is more work.

At first, I think I'll just output the following metrics after the download is done: Size of the downloaded file, time to took to download it and size / time. Its not as engaging as something approaching a derivative in real time, but for a first iteration to measure performance, I think it is good enough.

After that (once we have a measurement), I need to figure out what to do to improve the download speed. The hoped for approach seems to be splitting the downloads, but there are various ways of doing that with various trade-offs.

1. Just use arbitrary breakpoints in range downloads

This approach is very forward compatible (ie, GOG.com is unlikely to break it, it is part of the http standard) and it allows for very custom download granularity (ie, you can decide you want 20MB, chunks, 50MB chunks, 100MB chunks, etc).

However, because the granularity is custom, GOG will no provide checksum values for the chunks so in the event of a bad download, it becomes impossible to know which chunk is bad (unless there is a size mismatch) and you got to restart the whole thing from scratch.

2. Use the chunks recommended in the xml body provided by GOG for the offline downloads

I've noticed that GOG provides an xml body with checksums for certain range queries, but I've yet to take advantage of it.

I have two reticences about it: The provided xml was malformed for at least one game (Torment if memory serves, there might be more) so there might be some games for which you can't use it. Also, I have the vague recollection that this xml was use in the now deprecated GOG downloader so I have no idea how well it will be supported by GOG.com in the future

3. Investigate the new Galaxy Endpoints

This will be very forward compatible, but I see 2 separate issues: I'm not sure if the cookie info I'm using will allow the full usage of those endpoints (I've noticed it works for some, I'm not sure about all) and also, I'm not sure if it is possible to retrieve the offline installers via those endpoints or if you can only get Galaxy installers

Anyways, no matter what I end up using, I think I'll first do a POC via the "gog-api download" command before I refactor the tool completely with a particular solution.