It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
avatar
mrkgnao: I'm not too familiar with the gog-resume-manifest.dat file. Sorry.

I checked just now the one in my gogrepo directory. It has today's date and lists 0 games, whereas gog-manifest.dat has 1989 games.
Yes, if I remember correctly, it gets cleaned up before the gogrepo.py process normally exits. My code exits before then and wipes the gog-resume-manifest.dat file before each update.

avatar
mrkgnao: Did you notice that 55 is more or less the delta between the number you get and the number you expect? I bet that's no coincidence.
That did occur to me as maybe being related, but it is actually 1 title too many at 55 instead of 54, so probably just a coincidence. I'd have to recreate to double check. We are looking at 1610 - 55 which does not equal 1556, one out ... due to that demo that got added today.
avatar
mrkgnao: I'm not too familiar with the gog-resume-manifest.dat file. Sorry.

I checked just now the one in my gogrepo directory. It has today's date and lists 0 games, whereas gog-manifest.dat has 1989 games.
avatar
Timboli: Yes, if I remember correctly, it gets cleaned up before the gogrepo.py process normally exits. My code exits before then and wipes the gog-resume-manifest.dat file before each update.

avatar
mrkgnao: Did you notice that 55 is more or less the delta between the number you get and the number you expect? I bet that's no coincidence.
avatar
Timboli: That did occur to me as maybe being related, but it is actually 1 title too many at 55 instead of 54, so probably just a coincidence. I'd have to recreate to double check. We are looking at 1610 - 55 which does not equal 1556, one out ... due to that demo that got added today.
You can test it by deleting (or renaming) gog-manifest.dat and seeing whether it changes anything.
Post edited August 19, 2022 by mrkgnao
avatar
mrkgnao: You can test it by deleting (or renaming) gog-manifest.dat and seeing whether it changes anything.
Yes, I have begun think of doing that myself, as it isn't looking good right now, because 16 out of 17 so far that are in the gog-manifest.dat file are not in the gog-resume-manifest.dat file.
Yep, apart from the one exception, none of the games in the gog-manifest.dat file were in the gog-resume-manifest.dat file.

And renaming gog-manifest.dat file did the trick, and I got all 1610 titles.

Not sure what has changed at GOG that did this, but clearly also related to newer code in gogrepoc.py.

Something is now making it skip the entries listed in gog-manifest.dat, with one exception.

I guess the solution for my program is easy, a temporary renaming of the manifest during the update process.
Post edited August 19, 2022 by Timboli
avatar
Timboli: Yep, apart from the one exception, none of the games in the gog-manifest.dat file were in the gog-resume-manifest.dat file.

And renaming gog-manifest.dat file did the trick, and I got all 1610 titles.

Not sure what has changed at GOG that did this, but clearly also related to newer code in gogrepoc.py.

Something is now making it skip the entries listed in gog-manifest.dat, with one exception.

I guess the solution for my program is easy, a temporary renaming of the manifest during the update process.
I'm glad that's resolved.
BIG THANKS for your feedback mrkgnao, it certainly helped. :)
avatar
Timboli: BIG THANKS for your feedback mrkgnao, it certainly helped. :)
My pleasure.

Reminds me of a story I read in some old book about software debugging:
At some university in Australia in the 1980's, the computer support (a.k.a. IT) department was always flooded with issues from students and faculty. Eventually they found a solution. They put a teddy bear at the entrance to the computer support office. Anyone who had an issue had to tell the details to the teddy bear first, and only if that did not resolve the issue, could they approach the actual support staff. This immediately reduced the department's workload by more than 50%.

Glad to have been your teddy bear.
I just had a peak at the gogrepo change, and it's just a change already present in gogrepoc (it fixes the expected md5 path).

I'm not sure what's going on with your lists *hmm*, I'm currently doing something that prevents me plugging in my external hard drive for the next day or so but I'll have a look at what happens with my manifest when I can. It *might* be hidden games maybe ? I think I changed a default at one point so games hidden on GOG don't go into the manifest by default (there's a flag you can set to swap that back).

The resume manifest is too a first approximation just a list of games that need updating and who haven't had the full details of their update put into the manifest *and* saved, because you clear the new/updated flag on GOG whenever you "touch" the game rather than the shelf and the manifest only saves after a game's entry is fully updated. So games that don't indicate being new/updated will not end up in the resume manifest on a default run (apart from if it's the first run because that always builds a full manifest). It's maybe also possible that GOG changed the game slug or id but didn't mark the game as updated, that would cause things to desynch a bit until the next full update (I kind of assume that a full manifest update is done every 1-2 months.)
Post edited August 20, 2022 by Kalanyr
avatar
mrkgnao: My pleasure.

Reminds me of a story I read in some old book about software debugging:
At some university in Australia in the 1980's, the computer support (a.k.a. IT) department was always flooded with issues from students and faculty. Eventually they found a solution. They put a teddy bear at the entrance to the computer support office. Anyone who had an issue had to tell the details to the teddy bear first, and only if that did not resolve the issue, could they approach the actual support staff. This immediately reduced the department's workload by more than 50%.

Glad to have been your teddy bear.
Ha ha ha, yep better than being heard talking to myself.
avatar
Kalanyr: I just had a peak at the gogrepo change, and it's just a change already present in gogrepoc (it fixes the expected md5 path).
Goodo, I haven't checked it myself, and did wonder whether he might have implemented some kind of mrkgnao magic.

avatar
Kalanyr: I'm not sure what's going on with your lists *hmm*, I'm currently doing something that prevents me plugging in my external hard drive for the next day or so but I'll have a look at what happens with my manifest when I can. It *might* be hidden games maybe ? I think I changed a default at one point so games hidden on GOG don't go into the manifest by default (there's a flag you can set to swap that back).
Probably not relevant as I don't have any hidden games in my library.

avatar
Kalanyr: The resume manifest is too a first approximation just a list of games that need updating and who haven't had the full details of their update put into the manifest *and* saved, because you clear the new/updated flag on GOG whenever you "touch" the game rather than the shelf and the manifest only saves after a game's entry is fully updated. So games that don't indicate being new/updated will not end up in the resume manifest on a default run (apart from if it's the first run because that always builds a full manifest). It's maybe also possible that GOG changed the game slug or id but didn't mark the game as updated, that would cause things to desynch a bit until the next full update (I kind of assume that a full manifest update is done every 1-2 months.)
Yep sounds like a reasonable set of possibilities. And no doubt some perfect storm combination, as just prior to that, your latest version of gogrepoc.py seemed to be working fine.

I've coded a fix in my GUI now, using a 'Do Not Use' folder for temporarily relocating the manifest file. Which seemed a more elegant and safer way instead of temporarily renaming. There is also a program startup check to be doubly sure, though I doubt it will ever be needed.

Just about to upload my latest GUI update.
Haven't batch-downloaded my stuff for ages (well, ages during which haven't bought anything, this explain that), and so, back to this stuff. Which... ceased to work, on my end. So, let's try to summarize the previous episodes :

There is "gogrepo" AND "gogrepoc" ? Two tools doing the same stuff ? One being more functional or more uptodate than the other ? (I've been using gogrepo, and that's what I re-downloaded)

There's a recaptcha in the way now ? Apparently I'm not logged anymore, but it refuses to log me back on, because of that. Well, first the login functons gets me a
Warning (from warnings module):
File "[yes yes, hello there]\AppData\Local\Programs\Python\Python38\lib\getpass.py", line 100
return fallback_getpass(prompt, stream)
GetPassWarning: Can not control echo on the terminal.
Warning: Password input may be echoed.
Password:
Which I ignore, to be then informed that
cannot continue, gog is asking for a reCAPTCHA :( try again in a few minutes.
And it seems this has been happening to a lot of people. The only solution is cookie surgery, as I read here and there, or has this been solved at some point ?

I'm no not uptodate, on so many levels. :-/
avatar
Kalanyr: The resume manifest is too a first approximation just a list of games that need updating and who haven't had the full details of their update put into the manifest *and* saved, because you clear the new/updated flag on GOG whenever you "touch" the game rather than the shelf and the manifest only saves after a game's entry is fully updated. So games that don't indicate being new/updated will not end up in the resume manifest on a default run (apart from if it's the first run because that always builds a full manifest). It's maybe also possible that GOG changed the game slug or id but didn't mark the game as updated, that would cause things to desynch a bit until the next full update (I kind of assume that a full manifest update is done every 1-2 months.)
I have an elegant solution inspired by the principle of isolating pure processing from impactful actions in functional programming.

Basically, I generate an update file that reflects what GOG indicated has changed in the library. From there, you can update your manifest from this file any number of times you wish in an idempotent way.

Also, separately from that, the manifest update/generate commands create a state file indicating their progress (with the list of games remaining to be updated in the manifest) and when the resume command is invoqued, this file is read to figure out what is left to update.

avatar
Telika: And it seems this has been happening to a lot of people. The only solution is cookie surgery, as I read here and there, or has this been solved at some point ?

I'm no not uptodate, on so many levels. :-/
As far as I can tell, the only solution that we have so far is:
- Process the login page as a text file and circumvent the recaptcha (which works most of the time, but not always)
- Process the login page as a user interface in a full blown browser engine

The second option seems like it should work reliably, but is harder to achieve it in a multi-platform way.

So, this tool went the text processing path (works most of the time, sometimes doesn't, it didn't for me), I believe LGOGDownloader is going with the full blown browser engine (but only works on Linux) and my tool goes with cookie surgery (works reliably, works multi-platform, a bit of a pain to setup for your regular user though).

Choose your poison :P.
Post edited September 03, 2022 by Magnitus
avatar
Magnitus: I have an elegant solution inspired by the principle of isolating pure processing from impactful actions in functional programming.

Basically, I generate an update file that reflects what GOG indicated has changed in the library. From there, you can update your manifest from this file any number of times you wish in an idempotent way.
Not sure why you mentioned pure functions in functional programming as this is a massively IO-dependent operation. It's clear that gogrepoc is not a wonder of programming and there are improvements that could possibly be made, but it has worked for a long time and it has been maintained to a decent standard for that time, thank you again Kalanyr.

Really not sure why you mentioned idempotency either, I have never had files download multiple times for no reason, which would be the most annoying non-idempotent action. Everything else might break due to any of GOG not maintaining their metadata, or changing game ids, or their authentication system.

Please post your solution, and if it is better, and not too much inconvenience to switch, I will use yours.
avatar
lupineshadow: Not sure why you mentioned pure functions in functional programming as this is a massively IO-dependent operation. It's clear that gogrepoc is not a wonder of programming and there are improvements that could possibly be made, but it has worked for a long time and it has been maintained to a decent standard for that time, thank you again Kalanyr.

Really not sure why you mentioned idempotency either, I have never had files download multiple times for no reason, which would be the most annoying non-idempotent action. Everything else might break due to any of GOG not maintaining their metadata, or changing game ids, or their authentication system.

Please post your solution, and if it is better, and not too much inconvenience to switch, I will use yours.
I'm a fan of functional programming (not hardcore, but definitely an enthusiast) and I'm eager to extent its philosophy to things like operations (ex: limiting state sprawl) and things that are not 100% pure functions (in essence, making what is inherently impure behave in a purer way). This is something that I feel is underexploited and that really lights me up.

Otherwise, I'm very satisfied with the way this problem was tackled in my tool and am happy to share the idea. Maybe it will be implemented in this tool or maybe it won't, but I think the idea is worth sharing.

Concerning your last point, I'm not an extremely competitive person by nature, at least not in an extroverted way (I strive to be extremely competent, I don't enjoy selling it). I enjoy trying (not always succeeding, but always trying) to make fine useful things and I enjoy sharing some of the insights that gives me (sometimes comparatively by doing differentials with similar solutions that try to solve the same problem). I'm not here to convert you, merely discuss some of the insights I've had. You do what you like.

You're happy with this tool. It does the job. Please, keep using it.
Post edited September 04, 2022 by Magnitus
So, back in August 10, I asked if there were any updates for gogrepoc. What warranted that question were XML errors and allocation errors I'd get often, but I think I managed to solve that, at least mostly, and without updating the script. Probably something was broken in either my Python 3 or pip installs, or maybe both, or maybe something else entirely. Now, formatting my system and reinstalling everything required by the script seems to have solved the issues.