It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
high rated
http://api.gog.com/products?ids=2065676798

Above we have the REST API link for one of the more recent games released on GOG. What's there in the rest of the 2065676797 possible ids? Which areas are inhabited and which are deserts devoid of all life? One can only speculate (actually, there's a rough idea floating around already, but no hard facts).

Well, or one can write a mapping script and see what's out there. Which is what I did.

Currently got up to 36 million (that's id 36000000 to be more precise) out of the approximately 2065 million wide id range, but I'm determined to check each and every one.

Got a lot of help and input from SCPM, who I'd like to thank for explaining the finer points of the APIs I'm invoking.

There's also this thread which has proved immensely useful and covers stuff well beyond what I needed for my grand REST API expedition.

I plan to share my findings in some form once I return from the unknown, to whomever finds it interesting - it may take a couple of weeks though, as I don't want to generate too much load and bring the GOGBear out of his den.

Onward we ride!
Post edited May 12, 2020 by WinterSnowfall
Wonder if this will force GOGs hand to either reveal some of their plans... or shutdown the attempt...
avatar
Bigs: Wonder if this will force GOGs hand to either reveal some of their plans... or shutdown the attempt...
They've already hidden the APIs for upcoming / WIP or pending releases (you'll notice there's a "is_secret" flag returned for each entry used to implement that), so there are no leaks. All that I'll be able to see are already available games.

As I said, it's a charting effort, not a search for El Dorado.
Post edited January 18, 2017 by WinterSnowfall
My mistake
I've gotten down to the first 150 million ids... so far it's been quiet: except the first 10 ids which have some older (or better said ancient) games linked to them, perhaps only for testing purposes, everything is blank.

This fits into what I expected, anyway - I suspect there won't be any data until I get to around 1 billion+. The range may be empty simply because GOG decided to encode games with 10 digit ids. Of course there's no official hint regarding anything of the sort, so I'll still check every inch along the way.

I'll get back to you in a week or so.
500 million scanned, still nothing. The 10-digit id hypothesis may soon be grounded in fact.
Had to pause my scans for a week, but I'm back now and I've reached 1 billion ids. I can confirm there's absolutely nothing mapped between id 10 and id 1 billion - it's a great desert devoid of GOG life.

Seems like the 10 digit hypothesis is confirmed, at least for the time being.

But now I expect things to get really interesting in the range of the next 1 billion + ids.
So, I'm out of the desert and I've struck a vein: http://api.gog.com/products?ids=1094368489 is the lowest populated 10 digit id at the moment. I'm now getting one to several hits per million ids. Exciting times.

Had to stop the scan for a while and rework my script a bit - some Unicode issues (which were not a problem before Unicode values for various supported languages stared showing up) that have now been sorted.

I've also got plans to expand the script a bit and make it store all its findings in a SQLite database. But I'll do that on an optimized second pass, after I know what ranges I can skip and where the good stuff is at.
I'm willing to bet that the product ID codes are not starting at integer 0 and going upwards or anything like that, but rather are an integer formed from bit fields that have some kind of meaning internally, possibly acting as a prefix with a suffix that is a unique identifier for the given product. One would have to gather as many IDs as are known and do a numerical study of them looking for product related patterns to see if there are any.

1094368489 == 0x413ABCE9

Not seeing any obvious pattern in that, but it's just one product ID also. :) Perhaps all public product IDs have the upper nibble set to 0x4 as a public visibility flag or something, and the rest is a unique ID randomly generated.

What I'd try is to generate a table of all known IDs in decimal, and zero-padded hex and binary to see if any bit patterns appear likely or suspect in order to try to narrow down the search space.
avatar
skeletonbow: One would have to gather as many IDs as are known and do a numerical study of them looking for product related patterns to see if there are any.

1094368489 == 0x413ABCE9

Not seeing any obvious pattern in that, but it's just one product ID also. :) Perhaps all public product IDs have the upper nibble set to 0x4 as a public visibility flag or something, and the rest is a unique ID randomly generated.

What I'd try is to generate a table of all known IDs in decimal, and zero-padded hex and binary to see if any bit patterns appear likely or suspect in order to try to narrow down the search space.
Interesting points. Well, I plan to extract everything that's mapped at the moment and share my findings with everyone once the scan is done, so all manner of analysis can be conducted on the dataset once when it's ready.

I've also thought about how new ids are being allocated in the id space, and I'm also curios to see if GOG has somehow developed a mapping/hashing algorithm on a 10-digit output space based on a game's entry name, type etc. and if anything can be anticipated in this regard.

From what I've seen so far, it doesn't seem to be hex-based, at least nothing I've been able to catch on, but I agree we need more data before we can draw any conclusions.
Post edited January 31, 2017 by WinterSnowfall
avatar
skeletonbow: I'm willing to bet that the product ID codes are not starting at integer 0 and going upwards or anything like that, but rather are an integer formed from bit fields that have some kind of meaning internally, possibly acting as a prefix with a suffix that is a unique identifier for the given product. One would have to gather as many IDs as are known and do a numerical study of them looking for product related patterns to see if there are any.

1094368489 == 0x413ABCE9

Not seeing any obvious pattern in that, but it's just one product ID also. :) Perhaps all public product IDs have the upper nibble set to 0x4 as a public visibility flag or something, and the rest is a unique ID randomly generated.

What I'd try is to generate a table of all known IDs in decimal, and zero-padded hex and binary to see if any bit patterns appear likely or suspect in order to try to narrow down the search space.
The first few were actually just numbered sequentially from 1 (or 0?) (that is, not in the order they were added, but in the order the package was ready to add, even if perhaps the agreement was to add the game at a later date, or the agreement had not yet been signed), some have been removed (and perhaps re-added with a new ID, such as the Fallout games), but there are a few left in the 0-10 range that are still there since the first days.
Post edited January 31, 2017 by Maighstir
avatar
Maighstir: The first few were actually just numbered sequentially from 1 (or 0?), some have been removed, but there are a few left in the 0-10 range..
Ids 1 through 10 are still populated, but then there's a vast nothingness up to id 1094368489, as I've mentioned above. I think the first 10 have been left in their current positions mainly for debugging/testing and wouldn't count them in when doing any analysis of the overall id spread.
Post edited January 31, 2017 by WinterSnowfall
The latest game ids are all 2 something now (well, as far as I've noticed). I think it's 24 but I can't remember.

A lot of this data is probably already available through Magog you know... Hell I can give you a list that should contain all the known game ids if that helps...
avatar
Maighstir: The first few were actually just numbered sequentially from 1 (or 0?), some have been removed, but there are a few left in the 0-10 range..
avatar
WinterSnowfall: Ids 1 through 10 are still populated, but then there's a vast nothingness up to id 1094368489, as I've mentioned above. I think the first 10 have been left in their current positions mainly for debugging/testing and wouldn't count them in when doing any analysis of the overall id spread.
I would assume that they just decided to try obfuscating the numbers by starting at a suitably high number, then not increasing with 1 for each game/dlc/product/etc but increase with something like rand(1,1000) so that we wouldn't figure out if there are games ready to launch but not yet coming (say games 1-11 are available, and then game 13 is added, "hmm," says the community, "wonder what 12 is") or perhaps games that were ready to launch, but the publisher changed their mind, the community expects "game 12" to come soon.
Post edited January 31, 2017 by Maighstir
avatar
adaliabooks: The latest game ids are all 2 something now (well, as far as I've noticed). I think it's 24 but I can't remember.
Some of the newer releases have ids starting with 1 as well, if I remember correctly, so I don't think the allocation to the 1st billion + ids is done and dusted.

avatar
adaliabooks: A lot of this data is probably already available through Magog you know... Hell I can give you a list that should contain all the known game ids if that helps...
Thanks. I'm aware some/most of what I'm collecting is already available somewhere else.

I know about Magog and the treasure trove of data that I could tap into :)... what I'm doing with my scripts is more of a personal scripting exercise (have to refresh on my skills a bit) with the side-effect of potentially producing some interesting data dumps. I'm not trying to reinvent the wheel, but I need to write some customizable pythonic REST interfaces and couldn't think of a better way to do it (and test it :).
Post edited January 31, 2017 by WinterSnowfall