so how to make this work?

so how to make this work?

You need to wait a couple of days or apply the temporary patch floating around yourselves. I've got some time sketched in to fix this and test it on Sunday my time. Though if things go surprisingly well I might get to it Saturday.

I have finished updating my own program to cope with the errors on GOG's CDN and log them.

My current code does it this way: When a request doesn't work it retries up to 5 times with 3s in between. Only when all 6 requests fail the program will give up and log the problem.

So, yesterday evening I let it run through my whole collection and the result can only be described as devastating.

A total of 92!!! XML files were problematic, of which 37 were defective (probably all empty or 1 character), 50 could not be downloaded (secure connection failed) and 5 were simply missing (404 error). That's about 1.2% of the total number of files.

Does anyone know how (or to who) this can be reported? I doubt 1st level support would even know what you are talking about when you present them with a list of faulty XMLs.

Yeah, I tried to retry strategy too, but unfortunately, while the problem appears to vary across different parts of the cdn (and in some cases get fixed over time in the same part of the cdn), the errors are consistent in the same part of the cdn in the time frame it takes to update a manifest. If you'd be willing to retry over the course of days, maybe you'd see a difference in some cases.

I'm not sure how much official recourse there is.

The download links themselves, you can report, because they impact regular end-user experience in the browser, but the .xml metadata essentially points to what I think it s a legacy feature that is no longer used in the regular end-user workflow.

This is the problem with using things that are unofficial. They can break and as long as the official workflow keeps working, you're unlikely to get much help (ie, it is not officially supported).

This is why I think we really need an official api for clients. I understand why GOG is mum about it (I'm not sure they want to go there, supporting an official api is a commitment), but it certainly makes our lives needlessly complicated.

You might want to consider adding a flag that allows users to tell gogrepoc to ignore MD5/XML errors during download and continue as normal, as if MD5 check was correct (with just a warning). In case your fix doesn't cover all of GOG's failures. At the end of the day, MD5 is a nice-to-have, not a must for downloading files.

Yeah, I just quickly whipped up a retry method that repeats the XML collection for the files that failed it before and it managed to get 24 of the 92 failed files on a run just now. I'll keep trying to run it a few times over the course of the next few days.

I am also afraid that there is nobody that can be reported to. Maybe one of the blues could tell us more?

I also agree that a proper API would make the whole process far more reliable and would also be simpler to use. And it would also be easier on the GOG servers. Imagine you could get the file details for all files in your account (including their md5) with one single GET instead of having to do >15000 ...

Are there any proposals on community wishlist we can add our votes to?

Quite a few, actually.

https://www.gog.com/wishlist/site#search=api

Anything more specific to our goals with gogrepo?

Looking through that list several things come to mind.

1) Some are specific to a certain API, Achievements, and Cloud Saves. It doesn't look like these have anything to do with keeping my library up to date.

2) Is there something wrong with my web browser? I have difficulty understanding the purpose of a feature because visible HTML code is making it hard to read the description as well as some of the replies.

3) I suppose a feature made perfect sense to the person making the proposal but many of them don't convey that to the reader. This leads me to a theory about why these features don't get much attention.

A) It's not clear what they're asking for. I have some limited experience with programming. My educated guess is that the API describes how information is requested and how it's provided.

B) It's not clear why they want what they're asking for. What problem or limitation does the feature address?

C) It's not clear how the feature is beneficial. What will adding this feature allow users to do with it?

D) How can the request be fulfilled. When is the feature considered to have been provided?

I suspect other users don't know why they should vote for it, and the support team at GOG don't have enough information to act on.

Exactly that's what it is about. A Web-API is typically accessed with an URL that has specific data in it (either in the path or in parameters) that tells the API what data you want to have returned and returns the data you wanted in a well defined machine-readable format like JSON or XML. While you can view that data structure in a browser such URLs are typically accessed with programs or scripts that interpret the data and do something with it (e.g. store it in a database or format it in a way that is easier to read for humans and display it, etc.).

Imagine for example a weather api that you provide with the name of a city and it returns the current weather and a forecast for the next few days.

Like https://weatherdbi.herokuapp.com/data/weather/vienna which returns:

{
"region": "Vienna, Austria",
"currentConditions": {
"dayhour": "Friday 8:00 PM",
"temp": {
"c": 6,
"f": 42
},
"precip": "0%",
"humidity": "39%",
"wind": {
"km": 13,
"mile": 8
},
"iconURL": "https://ssl.gstatic.com/onebox/weather/64/partly_cloudy.png",
"comment": "Mostly cloudy"
},
"next_days": [
{
"day": "Friday",
"comment": "Partly cloudy",
"max_temp": {
"c": 8,
"f": 47
},
"min_temp": {
"c": 1,
"f": 34
},
"iconURL": "https://ssl.gstatic.com/onebox/weather/48/partly_cloudy.png"
},
{
"day": "Saturday",
"comment": "Partly cloudy",
"max_temp": {
"c": 8,
"f": 46
},
"min_temp": {
"c": 0,
"f": 32
},
"iconURL": "https://ssl.gstatic.com/onebox/weather/48/partly_cloudy.png"
},
{
"day": "Sunday",
"comment": "Partly cloudy",
"max_temp": {
"c": 9,
"f": 48
},
"min_temp": {
"c": -2,
"f": 28
},
"iconURL": "https://ssl.gstatic.com/onebox/weather/48/partly_cloudy.png"
},
{
"day": "Monday",
"comment": "Partly cloudy",
"max_temp": {
"c": 5,
"f": 41
},
"min_temp": {
"c": -4,
"f": 25
},
"iconURL": "https://ssl.gstatic.com/onebox/weather/48/partly_cloudy.png"
},
{
"day": "Tuesday",
"comment": "Sunny",
"max_temp": {
"c": 7,
"f": 45
},
"min_temp": {
"c": -3,
"f": 26
},
"iconURL": "https://ssl.gstatic.com/onebox/weather/48/sunny.png"
},
{
"day": "Wednesday",
"comment": "Partly cloudy",
"max_temp": {
"c": 9,
"f": 49
},
"min_temp": {
"c": -1,
"f": 30
},
"iconURL": "https://ssl.gstatic.com/onebox/weather/48/partly_cloudy.png"
},
{
"day": "Thursday",
"comment": "Partly cloudy",
"max_temp": {
"c": 8,
"f": 47
},
"min_temp": {
"c": -2,
"f": 28
},
"iconURL": "https://ssl.gstatic.com/onebox/weather/48/partly_cloudy.png"
},
{
"day": "Friday",
"comment": "Partly cloudy",
"max_temp": {
"c": 7,
"f": 45
},
"min_temp": {
"c": -2,
"f": 28
},
"iconURL": "https://ssl.gstatic.com/onebox/weather/48/partly_cloudy.png"
}
],
"contact_author": {
"email": "communication.with.users@gmail.com",
"auth_note": "Mail me for feature requests, improvement, bug, help, ect... Please tell me if you want me to provide any other free easy-to-use API services"
},
"data_source": "https://www.google.com/search?lr=lang_en&q=weather+in+vienna"
}

In all seriousness, I came to this thread *this month* (because I WAS in this thread before, I am 90+ % certain I even interacted with YOU of all people at some point before in this very thread) to seek help with an issue, ask about current state of things (as I was out of the loop for "a while") and perhaps contribute if I can.

I gave you the feedback that was not unreasonable.
Perhaps I didn't deliver it in the most precious delicately exquiside way possible but any normal coding person would have stripped any form and would have focused on the actual message. Because no matter the delivery the feedback is the most important part in the feedback message. That is if you care about the feedback in the first place.
You instead flipped out, and are continuing to flip out.
You are resorting to name calling me every other post when you respond to mine.
You call me this, you call me that, you accuse me of things that are unreal all while pretending that I have done some horrible horrible unheard of thing to you.

By the way (and I am fairly certain I told you this months ago), you are shilling your software (like every other page, all the time) a bit too much above the comfort zone of an average forum lurker.
My friendly advice is for you to tone it down a little.
But, of course, it is entirely up to you if you will even consider this particular feedback and if you will start spewing some BS about attacking you again because of it.

You are incredibly wrong in your judgments of my persona but unlike you I don't feel the need to drag this out in a spider-man pointing meme style.
I am too grown for that.

You got, let's face it, outraged because I "DARED" to comment on your little precious creation.
You can downvote me to hell all you want but from coding perspective your reaction is over the top bizzare.
If my feedback " "hurt" " you then I am sorry.
But you should imo change your attitude to feedback / commentary / criticism regardless of how fond you are of your software, what you think of people and yourself, what form the message was delivered in, and how much you do or do not care about opinions.

Plain and simple, accept the commentary silently and humbly, and consider it instead of pointing fingers and trying to pick fights, namecalling someone who gave you a legitimate feedback (regardless of the form, this was still feedback, and I spent some of my time that I am never going to get back in my life to review some of your repo and give you said feedback).

If you will keep this nonsense bizzare response up and going I reserve myself the right to ignore you entirely.

Btw (and yes, this is btw) don't ever shill your software aggressively to me again.
I am fairly certain I told you AGES AGO that I am not interested in it.

So like a seedbox?
You could maybe try with Amazon EC2 and the likes.
Not sure about the pricing (it would have to be calculated).

And I think it goes without saying that a service caching setup files for other gogrepo users to download from cannot be made in any capacity for legal reasons.
It would have to all be made one instance per one user (ergo every man for himself "if you're interested here's how", NOT "we setup a cache to speed things up, here it is").

And the idea only makes sense if GOG backend could *actually* deliver *dependable* connections and *sizeable* speeds, as so far with all the feedback across the internet it seems like their servers aren't all that fast and they may perhaps throttle connections after some time (and you can only hop IP's to circumvent it so much before you get internally flagged and or temporarily forbidden / account token banned).

What OS are you running this on?
This is just a thought but SOMETIMES (and depending on a system config too, tho that mainly applies to custom setups on Arch, Gentoo and the likes) dropping your caches mid-download may help (that is if the download agent, be it browser or else, throttles the download due to save device cache [or system maximum designated block device cache] clog) temporarily.
Do you have SWAP? (or pagefile if using Windows, tho the behaviour greatly varies between *NIX and Win in how they use it)

Perhaps we should instead ask for publicly viewable (and no login required) full documentation on Galaxy API and for somebody to then make a gogrepo-like script utilising it.
The API clearly is a thing, and it's said by GOG between lines that it supports all sorts of verify and download functionality that their web frontend doesn't.
The documentation for said API does not seem to exist in official capacity in a form that's usable.
Unlike Valve, GOG has a very covert approach here and doesn't seem very willing to make it public so far.
It would certainly make it easier to have it no dev account and no login required.

Update: While running the retry multiple times over the course of the last day while at the same time changing the CDN every time by jumping all over the world using Tor I managed to get all but the 5 missing XMLs. Those seem to be actually missing everywhere. And these files are:

/downloads/graveyard_keeper/en1patch3
/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/de1langpack1
/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/de1langpack2
/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/de1langpack3
/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/de1langpack4

The Graveyard Keeper patch is listed but the file itself doesn't exist anymore either. GOG obviously just forgot to remove the link.

The other 4 files are the German language pack for Witcher 3. While these files still exist they seem to be deprecated since the game now has separate installers for every language.

Thanks for this. I did something similar, but I changed the line to:

tmp_md5_url = response.url.replace('?', '.xml?') if "?" in response.url else response.url + '.xml'

That way it preserves the original functionality in the event that it actually gets what it thought it would, but it does the new thing if it doesn't get what it thought.

Hi folks,

Awesome thread here and on github...2 quick questions, using the vanilla script:

Q: After just downloading the manifest file there were some errors - the majority appear to have been resolved with the xml parser update. However the manifest is missing md5 data for a number of files.

1) Is there a "fast" way to just update those or do I have to update the entire manifest?

I didn't think to log the output (using tee https://forum.level1techs.com/t/guide-gog-game-file-archiving-howto/175857).

2) Is there as way to log any errors from within the script? I didn't see any.

Thanks

At least for me and a number of other people the Kalanyr's fork has less errors.
It's also actively worked on (contrary to the original that was forked from) so I encourage you to try it (WITH the manual fixes).
If you decide to do so I also personally encourge you to use the dev branch as it has the resume functionality (very useful for anyone with unstable network connection, also useful for squeezing that extra download in short free time).

https://github.com/Kalanyr/gogrepoc/tree/dev

The standard log logs *some* errors.
If you want to go hardcore full-on debug you can try running python with debug flags.
Tho beware it will likely produce ungodly sized backtrace.
I am also not 100% sure how to do that on Windows (I use mostly *NIX nowadays).

For vanilla:
1.Backup the original "broken" manifest (the gog-manifest.dat, tho you can also backup the resume one if you wish)
2.Move the original copy of the original manifest elsewhere
3.Look for the "broken" game in the original manifest
4.Get it's character string 'title' value (you can get this value from resume manifest too btw)
5.While leaving the resume file within the original directory run:
gogrepo.py update -id TITLE
( for example: gogrepo.py update -id the_evil_within )

That will produce a small manifest file with only that game info in it (assuming of course this will pass with no errors, which for some games seems to still not be possible even with the fixes).
Simply copy the relevant section from the small manifest and replace the corresponding broken section from the main manifest (note that you should do that on a working copy of said manifest and still keep the original untouched backup somewhere just in case something goes wrong).

For the Kalanyr's fork there's an option to use numerical 'id' value instead. You can also update more than 1 at a time (unlike on vanilla, at least according to vanilla's faq).
For example:
gogrepoc.py update -ids 1171957437 1083619800

edit:
fixed a mistake