Thank you for reporting this.

It's a MaGog bug affecting games that are not regionally priced AND are regionally blocked in Germany. There are four such games, all affected: Quake II, Quake III, Return to Castle Wolfenstein, and Wolfenstein 3D. The bug manifests itself only in the JSON output because the HTML output does not bother to display regional prices for non-regionally-priced games.

It should be fixed within 15 hours.

Until then, if you need to, you can manually edit the output to have DE list the same price as all other regions.

That was my first thought too. Thank you for chipping in.

No problem. I am having some problem with the "Vertical Bar Separated Values" and R, but I don't have a culprit yet. I will let you know when I have the time to look into it. I managed to work around it using a spreadsheet to export the file to CSV.
Thank you for working on this issue.

Here is a suggestion, if you don't mind: you might want to use a single representation for a missing value, such as "NA".

I believe I have seen "NO" for prices, and "Unknown" for dates. That is fine for human processing, but for automatic handling, we are always afraid that a value we have not seen before to represent an unknown or inexistent value will show up and make our code misbehave.

I am not sure what you are referring to. "NO" in prices stands for "Norway".

Generally speaking, MaGog's values are optimised for human display (i.e. HTML), since that accounts for more than 99% of MaGog's traffic. If I were to use some cryptic value such as "NA", I would have to replace it with, say, "Unknown", every time it has to be displayed, which is practically all the time. I prefer to put the burden on the uncommon case rather than the common one.

That said, I agree that consistency is a worthy goal. So, if you find some value other than "Unknown" or "", which are the ones I use most often, please let me know and I will consider changing it to "Unknown" or "".

You seem to be correct (as usual). I took a second look at the JSON. It does make more sense. (DE says nothing, other says NO... crazy!) :-D I'm sorry I jumped to conclusions without double checking.

About the HTML, I agree and I would do nothing different. JSON, on the other hand, is not for human consumption (even if your JSON is of the pretty type). And the CSV file should follow the JSON, IMHO.

I will try to think of something, and I'll let you know if I come up with something; but "translating" or "interpreting" values to display on the website might not be a bad thing, especially if you ever go international. ;-)

ANNOUNCEMENT:
Version 6.7.1 has been released with the following features:

* Fixed a bug that resulted in illegal JSON output for the prices in the DE region of flat-priced games that are also regionally blocked in Germany [reported by Gede]

Fixed

I think MaGog would be a more useful source of data to other projects if it made its JSON format easier to parse. I do not know if this is easy for you to do, or if you would be interesting in doing some changes to your system (maybe in a future major version) that would lead to the JSON and CSV-like representation to be more distinct; but the upside is that it would lower the barrier for someone to use MaGog's data.

For example:
- sale price history could be stored as lists with the pairs [date, discount] (not dictionaries);
- base price history could be stored as lists containing the triplet [date, previous_price, new_price];
- genres could also be lists;
- supported OS could also be lists; and
- bonuses could also be lists (it would be trivial to find out if the game had a manual or soundtrack);

I will need to study all the data with greater care, but this can give you an idea of what I mean.

Is it possible to add GOG Movies to search engine?

Thank you for the suggestions.

However, I am not sure we see the purpose of the JSON output in the same eye.

As I see it, the JSON output's purpose is to give other people access to MaGog's database exactly as-is (with zero data manipulation) to allow them to extend its functionality. The reason, for example, that genres/OS/bonuses/etc are strings and not lists in the JSON output is because they are strings and not lists in MaGog's database. And the reason they are strings is because I believe it is a more useful data representation than a list for these cases. If you wish I can explain why this is more useful. And trust me, nothing is trivial.

If you think that other data formats will prove more useful, is there anything that prevents you from writing a MaGog extension that translates from MaGog's data structures to your data structures? That is the whole purpose of exporting MaGog's data to JSON, to allow others to write extensions. So why don't you write it? If it indeed proves useful other people will use your extension, instead of working directly with MaGog's output, and everyone will benefit.

EDIT: If you wish, as an exercise, try to write the pseudocode for your trivial routine for finding manuals, assuming a list, and post it here. And we'll discuss it.

Yes. It is possible, but unlikely.

The format of movie pages is quite different from that of game pages and it would require a lot of work to add them. Given the small (and decidedly unchanging) number of movies, I do not feel it is worth the trouble at the moment.

I just want to see who wishlisted them

In order to do something like that, MaGog will have to build a database of which movies there are, what their IDs are, what their titles are, etc., which is the "lot of work" I discussed before.

What I might be able to do more easily is prepare you a list of people who have at least one movie on their wishlist and then you can check these very few wishlists manually yourself. Or if you're really interested in only one or two specific movies, I could probably tell you who has these specific ones without building a database. Would any of this help?

P.S. It would probably be useful to know what you need the information for.

Quick to reply, as always! Thank you, mrkgnao.

I will not insist on my suggestion, as you know better than anyone else what provides you with the best results. I can only guess how you store your data into your database.

Me writing a conversion routine was the obvious decision. I just thought of making it available to others in 1 less "hop". But it will be no problem to me, and little inconvenient to others (if anyone is interested in it, of course).

To acquire the full JSON I need to run a query on MaGog. I don't know if that causes any significant stress on your server, but static files are much lighter. Would you consider storing the last full JSON file alongside the CSV-like file at a fixed URL (maybe compressed to save on bandwidth too)?

I would appreciate very much if you could spare the time to share your opinion on this. I am always ready to learn.

Here is the full code in Python (Replace the __ with whitespace. This forum is not very Python-friendly):

import json
with open("magog.json") as magog:
__games_with_manual = [game["title"] for game in json.load(magog)["magog_search_results"] if "manual" in game["bonuses"]]

I'm sure you could write this in one or two lines in Perl ;-) In pseudo-code I could write:

FOREACH game IN gog_game_list
__IF "manual" IN game["bonuses"] THEN
____PRINT game["title"]

OK, so this code works on your JSON file "as is" (string) as well as on a list of strings, but if I wanted to do more complicated calculations, such as "what genres are more popular" or "what genre combination is more or less common", I'll have to decompose the string myself. The speed issue is not the most relevant here.

It is not too much trouble to do these kinds of changes. I'll just have to take a better look at all the information MaGog provides. There is a lot of information there.

Thanks for reading! Cheers!

I will consider doing that, but it probably will take a while since the code that generates the CSV-like is part of one application (the database updater that runs once every six hours), while the JSON export is part of another (the search engine). Until then, don't worry about the bandwidth and server load, but while testing your code you might want to work with a local file.

That said, the whole concept of the JSON export as part of the search engine is that it allows you to leverage all the existing capabilities of MaGog's search engine. If you want a list of all the games that have a manual, you don't need to get the entire database and code your own search engine, you just set the MaGog filter and get the results you want. Similarly, if you're interested in only say five fields, there no need to get the entire database, just set the display checkboxes to whatever you want and you have your results.

And, of course, you don't need the MaGog GUI to run the search as all the information is present in the resulting search URL.

You say "OK, so this code works on your JSON file "as is"", but it doesn't.

It will not find, for example, either of the Anno 1404 manuals because they are called:
- Anno 1404 manual (EN, FR, ITA, SPA)
- Anno 1404: Venice manual (EN, FR, ITA, SPA)

Actually it will not find quite a lot of the manuals in the catalogue. By a rough estimate, it will find 443 manuals and miss more than 480.

So you will end up with having to implement some more or less sophisticated form of string search, which you will use within your iterator, so what do I (as a search engine writer) gain from the iterator? Why not simply do a single string search? Which is what Magog does.

And what will happen when you come across an element called "manual avatars" because someone at GOG forgot to put the separator between the two bonuses (real life example, which has since been fixed).

Parsing these strings into individual bonuses, you will learn, is not that trivial, unless you assume the original data is perfect, which it is anything but. The trick it to write a resilient application that will yield useful information even with faulty data.

I agree with you that if you want to generate statistics (which is what you hint at) rather than search, you need to parse the strings into their separate elements and in that respect MaGog's output is less useful that the original GOG data (e.g. bonuses now have HTML separators in the GOG data (separators which Magog ignores), but when MaGog was written they did not). However, MaGog was designed as a search engine and it shows its design.

P.S. Do note that the "bonuses" string is not a comma-separated list. As you can see from the Anno 1404 example above a single bonus may contains commas itself.