On further exploration of the new catalog page, it looks like it is doing paginated queries to the following endpoint (example):

[url=https://catalog.gog.com/v1/catalog?limit=48&releaseStatuses=in:upcoming&order=desc:bestselling&productType=in:game,pack&page=2&countryCode=RO&locale=en_US&currencyCode=EUR]https://catalog.gog.com/v1/catalog?limit=48&releaseStatuses=in:upcoming&order=desc:bestselling&productType=in:game,pack&page=2&countryCode=RO&locale=en_US&currencyCode=EUR[/url]

I haven't seen this one before, but it does look less of a pain to work with in order to parse upcoming/newly added games. Since the old AJAX endpoints continue to work just fine I'll keep using them for now, but "behold the future" :P.

Edit: Sorry about the link above... welcome to the GOG forums!

I'm not sure what is going on as of late, but GOG appears to be blocking my IP for some reason. I can't access any of the API endpoints or even the forums any more (i get 403s every time)....... I had to use a VPN to get these few lines across. The main site appears to work fine, it's just he forums and APIs that are affected.

In case anyone was wondering why I'm not around, and since this appears to be the fate that the GOGBear overlords have in store for me, I may not post all that much any more (not that I have a choice). Daily driving a VPN service just to login to the GOG forums is not really something I'm willing to do.

I'll try to get any interesting stuff across when I can, as usual, but otherwise it's so long, folks. At least until the IP ban is lifted or whatever this issue is gets fixed.

It may be a new ban/throttling system is in place, I have no idea what's going on...

Edit: Well, whatever THAT was, it looks like it sorted itself out. I thought it might have been an issue with my ISP (due to other ne'er-do-wellers in my area, sometimes my public IP gets flagged as malicious), but the last few days only GOG has been acting up... weird. Anyway, now that this episode of "The Twilight Zone" is over, it's back to business as usual :).

*** Useful stats UPDATE ***

Most of my old public folder links are now obsolete (some may still be accessible, but I no longer own the account and it has been flagged for termination), so I'll reshare more permanent links here (Google drive will be around and free for the next few hundred years, I hope), in this post:

ID list: https://drive.google.com/file/d/1LNXLX-jc2uasingMzXTXOnTCVUiaauUj/view?usp=sharing

Deveoper/Publisher list: https://drive.google.com/file/d/13itDMZy46NwNm7dF-mEwvPHaKdpTQ4jj/view?usp=sharing

And, last but not least, the latest Airtable Galaxy vs offline installer version discrepancy link has not changed, but I'll also include it: https://airtable.com/shrldLsErlUf3eHqS

I think the base premise is broken. IDs are often incremental in DBs. A huge id field can exist, and is a problem for bits more than an indicator of a lot of entries. In some DBs you can't (or can but with some effort) reuse IDs so they tend to grow way faster than the objects you add to the list.

I went through the entire galaxy vs. offline airtable and evaluated each as best I can, splitting them into four categories.

1) FX = Fixed
Recently fixed by GOG. I expect these will disappear on the next airtable update.

FX 2021-09-03 1136126792 Absolute Drift osx 1.0g1 1.0g (c)
FX 2021-10-02 1301888900 Odd Realm windows 0.10.0.21 Production 0.9.0.13c
FX 2021-11-19 1441039809 X: Tension windows 2.3 1.0
FX 2021-12-09 1769244723 Wolfstride windows 1.0.6 1.1.1
FX 2022-02-01 1942260021 Afterparty windows 1.4.26f2 1.4.9
FX 2022-02-01 1942260021 Afterparty osx 1.4.26 1.4.9
FX 2022-03-10 1637485057 Unavowed osx 1.31 1.3a (GOG)
FX 2022-03-12 1104699058 Highrise City Demo windows 1.5 1.3
FX 2022-03-12 1890740105 Inua - A Story in Ice and Time osx 1.0.0 1.0.2.1
FX 2022-03-12 2033799141 Subverse windows 0.4.0 0.3.201
FX 2022-03-13 1168832459 ?V: Rings of Saturn osx 0.475.2 0.473.4
FX 2022-03-13 1232664301 Airships: Conquer the Skies osx 1.0.23.7 1.0.23.5
FX 2022-03-13 1432253578 PowerSlave Exhumed windows 1.0.1625 1.0

2) HE = Human Error
What GOG calls a human error, resuting in outdated offline installers. I will
attempt to report these to GOG (again).

HE 2020-01-20 1118073204 Sphinx and the Cursed Mummy osx 2019.08.28 2019.03.26
HE 2020-06-23 2086192498 Indivisible osx 42940 42416r
HE 2020-08-13 1333227204 Double Kick Heroes osx 1.66.6027 1.66.6017
HE 2020-12-04 1449856523 >observer_ osx 20171102 20171014
HE 2020-12-26 1465732730 Wetlands osx 1.0 0.9.3c
HE 2021-03-20 1276221446 Altered Destiny windows 1.0 0.9.1
HE 2021-03-20 1382504295 XF5700 Mantis Experimental Fighter windows 1.0 0.9.1
HE 2021-03-20 1867297613 4th & Inches windows 1.0 0.9.2
HE 2021-04-10 1802257607 Aground Demo osx 2.1.1b 2.1.1a
HE 2021-04-10 1802257607 Aground Demo windows 2.1.1b 2.1.1a
HE 2021-04-16 2078946998 Close Combat: Modern Tactics windows 2021.4.20.1 2007.12.12.1
HE 2021-05-19 1955433774 Nongunz: Doppelganger Edition windows 1.2 1.01
HE 2021-07-09 1767144970 Ys IX: Monstrum Nox (Demo) windows 1.0.6 1.0.0
HE 2021-07-13 1238406305 Black Skylands windows v0.2.6 (ebeccfb195) 98506d3d01
HE 2021-08-20 1825786806 Quake II RTX windows 1.5 1.0
HE 2021-09-02 1422911599 Timespinner osx 1.033 1.031
HE 2021-09-07 2012631912 Rescue Rover Collection windows 1.01 1.0
HE 2021-09-28 1964276929 War for the Overworld osx 2.0.8f1 2.0.7f1
HE 2021-09-30 1928042073 Hedon Demo windows 2.1.2 Demo 2.0.0
HE 2021-10-30 1865815565 HordeCore: Training Ground windows 1.10 1.6
HE 2021-11-17 1718511445 American Hero windows 0.0.31 0.0.30
HE 2021-12-03 1116987509 Eastward windows v1.1.0c v1.1.0b
HE 2022-01-27 1618128477 Necromunda: Hired Gun windows 62662 61851
HE 2022-02-03 1503947102 Concordia: Digital Edition osx 1.2.2 1.2.0
HE 2022-02-03 1503947102 Concordia: Digital Edition windows 1.2.2 1.2.0
HE 2022-02-03 1963130333 Diplomacy is Not an Option Demo windows dno_2.0.6_demo dno_2.0.1_demo
HE 2022-02-09 1472462985 Din's Legacy windows 1.013 1.011
HE 2022-02-11 1207660263 The Shivah windows 2.1 2.0
HE 2022-02-11 2000476666 Songs of Syx windows 0.60.25b 0.60.25
HE 2022-02-14 1207659043 Resonance osx 2.3 2.2
HE 2022-02-14 1207659043 Resonance windows 2.3 2.2
HE 2022-02-14 1207662913 Blackwell Deception osx 1.4 1.3
HE 2022-02-16 1771589310 Disco Elysium - The Final Cut osx f9ca75e8 61ad72b0
HE 2022-02-17 1294162796 Fates of Ort osx 1.4.0b 1.3.1
HE 2022-02-25 2069785347 SHEEPO windows Sheepo1.1 Sheepo1.0
HE 2022-03-02 1679108862 Ghost on the Shore osx 1.1.4.8040 1.0.4.8006a

3) ID = In Development
The newer galaxy version is not in the main branch, but rather in a
beta/indev/production/experimental/etc. branch (as indicated by the "Branch"
field of the galaxy build being set to some string). Since GOG stated that
they do not release beta builds on offline installers, these should probably be
removed from the airtable. You can either improve your algorithm by not
listing games where the newest galaxy build has the "Branch" field set
(better), or you can manually remove the games listed below (less better).

ID 2020-12-04 1953257763 Gary Grigsby's War in the East windows 1.12.06 Beta 1.11.03
ID 2020-12-04 2143654691 Heroes of Hammerwatch windows 106_test2 104
ID 2020-12-12 1445170412 Stars in Shadow windows 38803 (12/13/20) 38774 (08/09/20)
ID 2021-11-23 1444308510 Hammerting windows build2-tdtd-06964af_tdtd-patch-windows-dev build3-tdtd-06964af_tdtd-release_candidate-windows-release
ID 2021-12-15 1871006055 Little Nightmares II windows 1165 5.8
ID 2021-12-23 1711765822 Timberborn osx 0.1.2.1-c1267e3-xgm 0.1.1.1-db764df-
ID 2021-12-23 1711765822 Timberborn windows 0.1.2.1-c1267e3-xgw 0.1.1.1-db764df-
ID 2022-01-09 1207659172 System Shock™ 2 osx 2.48-update2 2.48
ID 2022-01-21 1176525525 Blade of Darkness windows v94 v89
ID 2022-02-03 1091815649 Colony Ship: A Post-Earth Role Playing Game windows 0.8.215a 0.8.206
ID 2022-03-02 1207659172 System Shock™ 2 windows 2.48-update2 2.48
ID 2022-03-04 1589675197 Space Haven osx 0.14.2 0.14.1
ID 2022-03-04 1589675197 Space Haven windows 0.14.2 0.14.1

4) TY = TYPO PROBABLY
I suspect the two version (1.3 and 1.1.3) are the same, with the difference
only being a typo in the label of one or the other. I can't be sure, as I
don't own the game.

TY 2022-01-06 1536961491 Archvale windows 1.1.3 1.3

Appreciate your efforts here, mrkgnao, as always :). Especially the part where you liaison with GOG trying to get these problems fixed...

Interesting find - I was not aware of it, nor am I storing off-branch builds any differently at the moment. Will definitely look into implementing this, but a rework of how I store build data is needed first, so it will take a while.

I've not looked into this too deeply until now, but apparently there's an additional deficiency in my comparison routines - I am not restricting offline installer version checks to the English language, and apparently GOG has multiple cases where non-English versions of offline installers are added at a later date WITH A LOWER VERSION. Because why not...

Multiple improvements are soon to come. Probably in the coming weekend the discrepancy table will be a lot more accurate thanks to your feedback.

P.S.: The following games are only available in non-English languages and will no longer be tracked because of this, but I don't really consider it a problem:

Kajko i Kokosz 1720224179
Wolfenstein II: The New Colossus German Edition 1285433790
Anstoss 2 Gold Edition 1808817480
ANSTOSS 3: Der Fußballmanager 1886141726

Alright, I've improved my discrepancy detection logic based on mrkgnao's feedback and my own findings. As a result of the rewritten code I am:

-> no longer taking branched Galaxy build versions under consideration (devel, legacy, exotic-platforms, tests etc.)
-> only checking the latest "en" offline installers (see the post above for more details)
-> adding back some almost certain false positives, because these have the potential of screwing over Galaxy users, and I'd like to keep track of them and hopefully get them sorted at some point (you hear that, GOGBears?). Here's the full list of re-added discrepancies:
- Deus Ex™ GOTY Edition -> I suspect some Galaxy builds are not public in the API although they should be
- CONSCRIPT DEMO -> gross mismatch between Galaxy build and offline version (who knows why)
- Police Stories -> no idea what is happening here and why the mismatch is there as I don't own the game
- The Binding of Isaac: Rebirth -> again no idea what this discrepancy is all about...

Since I have no idea about what's going on with Archvale, I'm going to keep it listed for now.

I'm not seeing a (public) 1.1.1 build in the API, so this is still a thing, apparently...

I just want to add that Quake II RTX is merely labeled incorrectly on the downloads page. The link gives you a v1.5 offline installer, so the file is already there, it's just mislabeled.
Edit: Same thing with >observer_ and Wetlands for Mac.... I have a feeling there will be a few more like this.

Thanks for confirming, I've added manual exceptions for those entries... I'll try and retest what I own out of the list, although I'm quite certain most of the remaining entries are not in my collection.

The Binding of Isaac: Rebirth and Chained Echoes Demo are also not labeled correctly. I've been able to check everything except for the following titles that I don't own:
4th & Inches
Altered Destiny
Axis & Allies 1942 Online
Black Skylands
Concordia: Digital Edition
Eastward
Ghost on the Shore
Necromunda: Hired Gun
Nongunz: Doppelganger Edition
Onde
SHEEPO
Songs of Syx
Spirit of the Island
TUNIC
Wolfstride
XF5700 Mantis Experimental Fighter
The only other one I have a question about is Deus Ex Revision, because it appears the offline version is more up-to-date than the Galaxy version?

As I said above, I suspect some Galaxy builds which should be public are not in this particular case. I'm almost certain the right build entry is there, just not visible to us mere mortals :P.

Thanks for checking everything you could :). I've also gone through everything I own but nothing was off on what I checked and the discrepancies still stand.

Oh FFS...

Does anybody know what U+0097 is supposed to do and why GOG feels the need to include it in the product description of the following IDs (yes, it's on the product webpages too!)?

1100566473 - Drakensang
1202885143 - Tomb Raider: Anniversary
1438332414 - World in Conflict: Complete Edition
1809223221 - Crysis®
1948823323 - SPORE™ Collection

It will apparently trip python code into interpreting unicode text as byte arrays sometimes which blows up pretty spectacularly on otherwise perfectly valid code... thanks, GOGBears, you're awesome, as always :|.

Edit: As I've long known, there be monsters in the GOG APIs... further invesigation has surfaced various instances of U+0092, U+0093 and U+0094 casually sprinkled on changelogs and product descriptions........

I've written a little Unicode conversion program in AutoIt, a basic-like scripting language, and while I get back an ampersand for the likes of U+0026, I get nothing, not even a space for the ones you have listed.

So it may be no issue to just simply remove them from any return.

P.S. I use my code with descriptions and changelogs from GOG, and it just does an auto replacement with the conversion result, and I've not yet seen anything untoward ... not that I have specifically checked for those entries.