Oh boy, lookie here, a game entry without a title:
https://api.gog.com/products/1441272224?expand=downloads,expanded_dlcs,description,screenshots,videos,related_products,changelog

Note to self: put a lot less NOT NULL restrictions next time you design a table to store GOG API return values.

Now that I have the data in a database, it's a lot easier, at least in theory, to pull out relevant information. For example, here's a heavily filtered list of games that the GOG product APIs consider to be yet unreleased: https://dl.dropboxusercontent.com/u/1845258/gog_unreleased_02032017.csv

Even if it's filtered, there are still some erroneous entries for games which obviously were released a long time ago. What I've mostly filtered out are entries marked as "invalid", "to be deleted", "wrong game id" and stuff like that.

Most of them have been flagged already. Some of them are still mysteries. For example, I have no idea what this is:
[url=https://api.gog.com/products/1589615681?expand=downloads,expanded_dlcs,description,screenshots,videos,related_products,changelog ]https://api.gog.com/products/1589615681...[/url]

Is it perhaps just a test API entry for our friends at THQ Nordic? Any better ideas?

For anyone that's still tuned, just wanted to let you know that I've completed a second full scan of the entire product id range, now saving the API responses to a database along with extended id query data, including stuff like changelogs.

I've also written an update routine which allows me to keep all the mapped ids stored in the database in sync with the latest changes on GOG and also allows me to detect any removed or deleted entries.

I'll post an analysis of my findings soon enough. Watch this space.

Edit: A first notable thing I noticed is that they fixed the extended APIs for movie entries - you no longer get the GOGBear when including downloads as a field, and now it actually returns the right info!

I've now started a 3rd full products API scan after overhauling most of my scripts.

Among the changes are:
-> Re-defined how movie entries are identified, since GOG has fixed movie entries to include downloadables (I now do a description-based identification, since it seems to be the simplest way)
-> Save and include link information as separate DB fields
-> Better company match for authors/publishers - I will do an upper case match stripping various characters like ':', ",", "/" and "." as these tend to cause matching issues between what is listed on a product's page and the company values I parse from the gogData variable
-> Update run mode in which existing DB entries are scanned for changes and updated if required (also useful to detect changelog updates as well as incoming releases for yet unreleased games that have a visible API entry)
-> The update run mode can now identify when a previously public API entry is removed/hidden (this can happen, as I've seen with the Homeworld: Deserts of Kharak DLCs that I previously uncovered)

As always, I'll keep you guys posted if anything interesting shows up.

I've been looking a bit into how GOG allocates ids for new games. In short, I'm sticking to my original theory that they're using trained monkeys to pull (10) sequences of digits out of a top hat.

Seriously now, there has been some noise in the lower end of the id range, so that's an area to keep an eye on usually. Some of the new Delta Force games, for example, are there:

https://api.gog.com/products/1079289132?expand=downloads,expanded_dlcs,description,screenshots,videos,related_products,changelog
https://api.gog.com/products/1160509725?expand=downloads,expanded_dlcs,description,screenshots,videos,related_products,changelog

Full Throttle is a bit higher up:

https://api.gog.com/products/1325415523?expand=downloads,expanded_dlcs,description,screenshots,videos,related_products,changelog

And I'm still scanning for the rest... but there seems to be no preference or logic in how they are allocated to be honest.

It seems the throttling police is back on duty. Officer GOGBear started issuing tickets to everyone going above 20km/h on the GOG products API highway; the tickets include a nicely standardized HTTP 429 error code (Too Many Requests).

My scanning speed took a hit but, arming myself with a new level of python mastery (yeah, I feel like I leveled up), I was able to minimize the damage and still go 40km/h without getting a ticket (for a long period of time, if not forever)... regardless, the speed is still at least twice slower than it was before.

Gee, thanks GOG...

Good news, GOG has lifted all bans and limitations on its APIs!... yeah right, I only wish :).

The non April-fools truth is that I've spent more time coding in order to address GOG-imposed limitations in a not-so-time-taxing manner than I have on the actual data parsing. I'm not saying I'm not getting anything out of it, since I'm learning new stuff about threading/interlocking and inter-thread synchronization which I've never bothered with before, but it's still a bit frustrating to spend time optimizing work-arounds to all the GOG road blocks.

Open your API borders GOG, and punish the real service abusers! Stop this GOGBear police brutality!

I almost laughed my arse off when I found this one:
https://api.gog.com/products/1447152392?expand=downloads,expanded_dlcs,description,screenshots,videos,related_products,changelog

On this historical day I've finally completed a 3rd full products API scan. It was probably the longest scan of the 3, due to the new API throttling measures that the GOGBear has rolled out.

Some highlight of this scan are:
-> GOG has done an overall revision of their API entries to update some of the included product links from http to https - I believe this was done before the Galaxy 1.2 launch
-> I've detected no new yet unreleased & unannounced game API entries, a sign that GOG is being more careful with their API exposure (a good thing as far as I'm concerned, though it provides us less insight into the GOG release future)

If anyone wants a copy of my SQLite database with the data I've collected so far, please PM me, I'll gladly share it with you.

One more thing to mention is that this is probably the last full scan I will ever do and that I'm working on a system to automatically pick up new announced/released game ids to add them to my database.

As always, I'll keep you posted and will highlight anything that stands out while I'm keeping a watchful eye over the API landscape.

Ups, they did it again...

Old ids (now game ids):
https://api.gog.com/products/1458058109?expand=downloads,expanded_dlcs,description,screenshots,videos,related_products,changelog
https://api.gog.com/products/1454587428?expand=downloads,expanded_dlcs,description,screenshots,videos,related_products,changelog
https://api.gog.com/products/1454315831?expand=downloads,expanded_dlcs,description,screenshots,videos,related_products,changelog

New ids (now pack ids):
https://api.gog.com/products/1242989820?expand=downloads,expanded_dlcs,description,screenshots,videos,related_products,changelog
https://api.gog.com/products/1312824873?expand=downloads,expanded_dlcs,description,screenshots,videos,related_products,changelog
https://api.gog.com/products/1248282609?expand=downloads,expanded_dlcs,description,screenshots,videos,related_products,changelog

Apparently even the new Bethesda games will earn you the right to a free copy of Arena and Daggerfall. This was not the case until today.

There's a new products API return field in the "videos" section, namely "provider". A hint that GOG is branching out to other video hosting services perhaps?

Anyway, it was a bulk update on the db (I guess), so I'm seeing massive updates on all the product API entries because of this.

"videos": [
{
"provider": "youtube",
"thumbnail_url": "https://img.youtube.com/vi/R60v-ERygnA/hqdefault.jpg",
"video_url": "https://www.youtube.com/embed/R60v-ERygnA?wmode=opaque&rel=0"
}

Well, it's done and it seems to be working as expected... the only surprise I had is that the downloads section of the products API also includes info regarding the Linux installers (wohooo!). Turns out only the content-system APIs know nothing of Linux yet.

Now I'll have to check my system against what MaGog is reporting to see how accurate it is. Fingers crossed.

Massive updates today on all the ids, adding the following 2 fields in the "images" section of the API returns:

"menuNotificationAv":
"//images-1.gog.com/0ee7426dcc9c0be3ab629f72e7f8cda0c46b876ca9ebfb931e3d5356b4ce28be_menu_notification_av.png" ,
"menuNotificationAv2":
"//images-3.gog.com/0ee7426dcc9c0be3ab629f72e7f8cda0c46b876ca9ebfb931e3d5356b4ce28be_menu_notification_av2.png ",

Is this a sign that a long-awaited notification option is being rolled out?

Interesting, and with pictures of the actual games too..

Strangely enough there are some bundled games that are not exposed at all via the products API, for example:
https://api.gog.com/products?ids=2134842136

This should show an entry for Sudden Strike 3: Arms for Victory, since it's already been released part of the "Sudden Strike 3" bundle containing both Sudden Strike 3: Arms to Victory and Sudden Strike: The Last Stand, but for some reason it's not.

I also found a similarly not exposed/hidden entry for The Humans III: Evolution—Lost in Time..., and I'm sure there are plenty more similar entries there in the GOG catalog.

Needless to say this makes tracking bundled game content almost impossible, since only the bundle will be visible via the APIs.