For the past few months I've been getting an error in my gogrepo.log saying:
xml parsing error occurred trying to get md5 data for setup_a_golden_wake_2.0.0.4.exe

The file and it's other information download fine but the md5 is listed as None in the manifest.

Same here.

*sigh* Even though it saddens me a little to say this but it wouldn't be GOG if everything worked as it is supposed to.

I can't check that particular file since I don't have that game but can someone who knows how to get the XML post it here?

For that particular game, I think that error has been there for years.

Well, as long as the game files still download fine, and that is not spreading to lots of other games...

Which leads me to another question: How do you even know that a file was changed?

/account/gameDetails just lists download links reliably. Version can be null and date can be empty so using that data wouldn't work.

You'd have to get all the XML files and store the md5 for every file and you know that a file has changed when the md5 has changed (after all GOG did sometimes change files without changing the version, filename or filedate) but that relies on the md5 to be available and correct all the time which doesn't seem to be the case either according to HappyPunkPotato and ikrananka.

So what is your way of detecting changed files?

Most extras don't have the long MD5 data , and some games / files are missing them.

GOG marking a game as new / updated is the primary mechanic. After that it's file name (which generally changes), and file size. MD5s are only checked during verify (and chunks during download) because it actually takes a reasonable amount of time to calculate the MD5 of a 4 GB file (and even longer for the Linux installers which can be ~25 GB) so they aren't a practical quick check.

There's some more heuristics that you could do (in addition to the stuff you mentioned) like invalidating a file if what GOG says the MD5 is is different from what's in the manifest (for example).

Which reminds me I should probably double check that a file being marked as pre-verified for skip purposes gets cleared if it's reported MD5 by GOG is different from the manifest, I think it already does that but should make sure.

I've only been using the forked version for downloading for about a week.
In all that time, I have never been able to get a download speed above 1.8 Mb/s with it, and that is with 4 threads enabled, whereas with 1 enabled, the maximum speed I get is 1.3 Mb/s but usually less, much less on average.

With the original version of gogrepo.py, and using only 1 thread, I was often able to get 5 Mb/s or slightly more, though average was probably around 4.5 Mb/s for the downloading speed.

Has GOG been slowing things this last week, deliberately throttling, or is that lower speed, something I can always expect with the forked version?

I've been busy developing so I haven't really done extensive tests about this yet, like re-instating the original and testing how that is currently working. But I know that a third party program like Free Download Manager 5 can still download at around 5 Mb/s for me from GOG most of the time. But I imagine it is using more than one connection to do so.

I'm not a big fan of downloading multiple files simultaneously, certainly not as the default and not optional, but 4 connections (threads) for one file is fine. Can the forked version do that?

EDIT
I have just checked my settings in Free Download Manager 5, and when it is set to flat out (my default) it will do up to 5 connections if allowed, but only ever one file at a time. I don't know what GOG allow.

My download speeds have been normal, though Linux / Mac downloads are sometimes much slower, I think GOG simply has fewer servers and/or less bandwith allocated for those.

No, you need MD5 chunk data to download multiple threads for a single file safely and the original doesn't use that. I haven't extended the threading to take advantage of it (it's not super simple since the MD5 chunk data isn't always available and is sometimes wrong),

Mmmm I'm not sure what is going on then ... unless they have made a permanent change.
For now, I am just using FDM5 to get my downloads from GOG in a timely fashion, and then using gogrepo.py to verify or as part of my new Simple GUI program.

I will also get around to doing some more comparative testing.

Thanks for the info.

Of course, it wouldn't be GOG if all data were complete and consistent. :-(

I was looking for finding a logic that reliably tells me if a file has changed with as little complexity as possible. If possible even without downloading all XML files. Sadly up to now I don't see any method where I can do so without the XML files and even with the XML files the logic can not work 100% since there is so much missing data. :-(

GOG marking a game as updated is far too error prone to be usable. Someone is even keeping a statistic how many games have changed every week that have not been marked as updated.

By checking the MD5 I didn't mean to calculate it from the downloaded file but comparing the MD5 from the XML to the stored MD5 from the database. If the MD5 has changed the file has obviously changed. But if the MD5 is often missing then this method obviously doesn't work. :-(

Calculating the MD5 from the file content only makes sense after downloading to check if the download was successful or on user request.

I guess for the moment I'll settle for this logic: If any of filename, filesize, filedate or MD5 has changed then the file has updated. But that requires to download all the XMLs of course.

I guess that's what you are doing as well since I have read creating or updating the manifest files can take many hours.

BTW, how big can the XML files get?

Totally, so aside from MD5, which may be missing, the only other way to know of an update is filename, if that has indeed changed.

So not much we can do, if update flag hasn't been set, MD5 is missing, and file name hasn't changed.
That said, I guess the odds are in favor of one having been done at least ... one can but hope.

P.S. I forgot about file size, so perhaps that is a 4th check.

BTW, I keep hitting a certain problem that I have not found a solution for yet. The login procedure.

At the moment I'm doing it exactly by the book (or rather the unofficial API documentation):

First I get the page from https://login.gog.com/auth?... with the documented parameters, extract the login token from the page, and post username, password and login token to https://login.gog.com/login_check. On a successful login the result is a redirect to a specific url which contains a login code.

Then I request https://auth.gog.com/token?... with the documented parameters and the login code to get an OAUTH token which should be refreshed once an hour. I'm not doing that yet since my script is only doing some data collection right now that doesn't take very long. The OAUTH token contains among other things an access_token that should be used in the header of each request to the server. The header should be "Authorization: Bearer " + access_token.

Despite being quite convoluted it does work just fine once you've got that down.

However, when developing you often make changes and test them to see if they work as expected. That leads to many logins within a short time since every time I start the program the login runs from scratch. And it seems that GOG doesn't like that. After a while I get put on a blacklist which requires me to answer a captcha at login which obviously foils the automatic login. Then I have to wait for a few hours or sometimes even days until I'm off the blacklist again.


So the question here is: Is there any way around that? Can I somehow prevent being put on the blacklist? Or can I make the login persistent so that I don't have to login again and again? What about the OAUTH token? If I can make the login persistent I probably need to refresh it. How do you handle all that?

Speaking of update flags, I actually wrote to GOG support a while back asking about that and they just got back to me and claim it's been fixed for some time. :S

They asked if I can send a screenshot of it not working... I'm not sure how to prove the absence of something. Anyone got any good ideas about how to explain what isn't happening?

First and foremost, thanks for answering my questions. I hope I don't take up too much of your time.

Whenever? Even after a day or longer? The API doc gave me the impression that the whole OAUTH data expires after an hour which means that the refresh_token also becomes invalid. Or in other words that the refresh had to be done within the hour of validity to get new OAUTH data before the old expires.

If that is not the case and the refresh_token stays valid even after expiration of the access_token then I can skip the login if I have an old token that I can use to get a new one.

But that leads to several new questions: Do you know how long the refresh_token stays valid? Days, weeks, months? If I try to use the old refresh token to get new OAUTH data how do I know if that succeeded? Is the result I get on a failed attempt not a valid JSON file? Or do I have to check the HTTP status code (401 or 403)? Does an attempt to refresh OAUTH with an invalid or expired refresh_token put me on the blacklist?

Thanks for offering code but that's not really neccessary. The problem isn't producing code. The problem is coming up with a valid program logic when the server's behaviour isn't fully documented and the data the server returns is woefully incomplete.