Unreal Engine 4?

Yep, probably this. I was thinking maybe it stood for a game series I was not aware of, but this makes more sense. Thanks :).

I'm glad to report I've finished a new round of full ID range scans which yielded quite a few IDs I could not have found through any other means.

I've compiled 3 exports of my product DB for anyone who is interested in checking them out:
-> full export, with anything you might be interested in, including changelogs:
https://www.dropbox.com/s/tbqbmc3ew0950qd/gog_products_full_20180928.csv?dl=1
-> lite export, reduced field set, but also includes changelogs:
https://www.dropbox.com/s/1gimh67bturckwt/gog_products_lite_20180928.csv?dl=1
-> ultra lite export, which contains only ids and product names:
https://www.dropbox.com/s/guyek24b3nq1jo0/gog_products_ultra_lite_20180928.csv?dl=1

I can't even imagine how time consuming a complete scan would be, so take what follows from someone clueless. While the complete lists are great, could you also compile a list with only the latest findings? I'm actually curious about what were those "quite a few IDs I could not have found through any other means. "

Takes about a couple of weeks, but I'm a patient man :).

The full export also contains the date on which an ID was detected and added to my database - it's actually the first field in the list (gp_int_added).

Of course the stuff you're curios about is mixed with regular new ID detections, but you should be able to figure out which is which, because the more obscure stuff does not have a product page and therefore no linked authors and publishers.

P.S. What I meant by that was mostly some Fig and Kickstarter exclusive entries, plus some other IDs which were never linked to a store page, therefore slipped through my regular detection routines. If there were any truly interesting stuff to report, they'd show up on the "Games that are *unofficially* confirmed..." thread ;).

That's what I needed, to be able to identify what was new. Thank you! +1

Oh man, am I glad I took a backup of my DB yesterday... let's just say my scripts will need some rework due to the site changes. What a perfect opportunity to have a look over XPath and use a proper HTML parser in my code.

That and I'm also hoping some mismatches in product_urls introduced due to the changes performed today will be corrected sooner rather than later. For example, you will notice movies are now listed under www.gog.com/movie, but their API product_urls still uses /games :).

... and I thought yesterday was bad. Hello entirely restructured pages and gogData variables. Back to the drawing board for a few days, I guess.

Hmmm... ok, so the main homepage content section is, in fact, an iframe: https://www.gog.com/front_page

A weird structure this new site has (or so Master Yoda seems to think). A weirder data structure it employs... but my scripts seem to have adjusted to it. Now, if it could just stay still for a day or two.

Yes yes it is. I was modifying it last night and noticed it was an iframe which I also found weird.

And they are still working on the site it seems... my new XPath queries failed today due to an added class definition. T'was a quick fix, Horatio, but still annoying. When will our ordeal end?

The sentry GOGBears will be happy to know that due to personal reasons and time constraints I've drastically reduced the daily amount of data that I'm collecting, ergo the number of API calls I shoot out their way.

I'm sticking to company and products related data collection as a daily activity, since they are required for changelog and offline installers reporting, but everything else (Galaxy versioning and whatnot) I will most likely collect once every week or so from now on, probably over the weekend.

This has drastically decreased the amount of time required to complete my daily scan routine to about 40 minutes, down from about 2+ hours.

... so I've finally figured out a reliable way to get the new and upcoming game ids, which doesn't really have much to do with the front-end stuff (yey!). It's pretty obvious once you look at the page load flow, but I was pretty much overthinking the problem, as usual.

These nifty little APIs, which are intended to be run as AJAX queries when the game page loads or changes, can give you pretty much anything you're looking for in a nicely parsable JSON format:
https://www.gog.com/games/ajax/filtered?availability=new&mediaType=game&page=1&sort=date

No more Mr. "crawl through the HTML DOM and get stuff out of the gogData variable" guy.

But you know, GOG staff often forget to add something in the right place.
I will not be surprised if the new method failed to detect one or more new games.

Yes, you're right, it happens, but of all places this is the one they're least likely to miss. To address that I also run full id-range (exhaustive) scans every now and then.

Still, I expect these new endpoints will catch 99% of them, which is better than the parsing of the gogData variable I was doing previously. It seemed to work well enough with the old site, but the new one has changed the rules and a lot of things were starting to slip by me.

adaliabooks is also graciously exposing the ids picked up by people who use his scripts, which sometimes gives me insights into game ids which are part of a pack, that are never ever exposed on the UI, so there are plenty of fishing nets laid out there.