It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
avatar
_Auster_: So, back in August 10, I asked if there were any updates for gogrepoc. What warranted that question were XML errors and allocation errors I'd get often, but I think I managed to solve that, at least mostly, and without updating the script. Probably something was broken in either my Python 3 or pip installs, or maybe both, or maybe something else entirely. Now, formatting my system and reinstalling everything required by the script seems to have solved the issues.
It's a known issue since February 2022. It's entirely on GOG's side, nothing to do with your installation.
avatar
Magnitus: I'm a fan of functional programming (not hardcore, but definitely an enthusiast) and I'm eager to extent its philosophy to things like operations (ex: limiting state sprawl) and things that are not 100% pure functions (in essence, making what is inherently impure behave in a purer way). This is something that I feel is underexploited and that really lights me up.
Yeah for sure.

The future of programming is functional programming because if you want bug-free software with maximal verifiability then separating pure/impure functions is the first and most important step.

As much as I agree with you and would love to agree with you in a thread about best programming practices, I just think it's out of place here where the biggest problems/'bugs' with this tool is the change in functionality in the undocumented backend of GOG which seems to be an abandoned unofficial API that has stopped living up to its contract.
avatar
lupineshadow: Yeah for sure.

The future of programming is functional programming because if you want bug-free software with maximal verifiability then separating pure/impure functions is the first and most important step.
I think, if not taken to extremes, it is the better way to do software, but I'm less confident than you about it being the future of programming (not because it wouldn't be better, but because the world is imperfect).

I tried to ship functional programming at the previous two places I worked at with little success.

For a bit, I was hoping Javascript might be the first really mainstream language that veers in that direction (except for types, it was flexible enough to support it), but then Typescript became all the rage (probably thanks to devs coming from OO languages like Java or C# wanting something familiar to look at) and now the direction of the language is very much OO.

There are two kinds of software professionals: Those who learn and those who get married to whatever was trendy when they first learned their craft (just look at the significant number of developers still going apesh*t about php, that was hot back in the beginning of my university years circa 2002, but in the following 2 decades, some people learned jack sh*t about better alternatives... I loved php too back in 2002-2003, I've moved on).

Too many people in the second category learned OO in school. It will take at least a generation to unlearn that, but even so, existing momentum might make it longer.

Atm, the best I got for you, at least for servers, are microservices. If people write sh*t code (and lets be honest, a lot of people do), make the code less important by breaking it down into small individual codebases with clear api contracts that are easily replaced cogs in a big machine. Hopefully, less of those 100k+ lines monolith where you got to live with the god awful technological choices of the less able forever and ever.

My apologies if I sound demeaning, but I've seen a lot.

avatar
lupineshadow: As much as I agree with you and would love to agree with you in a thread about best programming practices, I just think it's out of place here where the biggest problems/'bugs' with this tool is the change in functionality in the undocumented backend of GOG which seems to be an abandoned unofficial API that has stopped living up to its contract.
I think discussing the principles of functional programming is rarely out of place, and I think its relevant to the particular problem that was discussed for updates.

Concerning the broader problem of the api, I'm really wondering if leveraging the Galaxy api might be the better alternative. GOG seems more eager to maintain that properly and my understanding, from what I could discern, you can leverage it to download offline installers too?

If so, I think the main issue might be the login: from what I recall from talking with Sude, cookies don't cut the mustard with the Galaxy api and you need some other token that is hard to obtain if you don't do a proper login.

I think at that point, you'd need a full blown rendering engine to extract that which is harder to achieve cross-platform (something that is of importance to me).

I started learning Rust recently and I'm contemplating whether I could possibly leverage this project to do it (seems promising): https://github.com/tauri-apps/tauri
Post edited September 05, 2022 by Magnitus
For anyone having reCAPTCHA errors, it is because of a bug occurring in the detection of it in the script. Some page update broke it. IF when login in through the browser you don't see appear it won't be a problem through gogrepoc.py.
Thanks to MattMills https://github.com/eddie3/gogrepo/issues/65#issuecomment-1250761919 for finding this out.
This affects both eddie3 and
Kalanyr (*the one I'm using) versions.

I've wasted 4 hours trying to extract cookies... what a mess theses are now, _abck ak_bmsc bm_mi ....??
Anyway

Find line:
if len(etree.findall('.//div[@class="g-recaptcha form__recaptcha"]')) > 0:

Replace 0 with 99999999 , save.

Login through browser, if no recaptcha then do the: python gogrepoc.py login
Post edited September 21, 2022 by Lostinablock
I've backported the token login to the current version of GOGrepoc which lets you use a browser to bypass the ReCaptcha. I'm currently doing a local test (seems good so far) and am looking for some volunteers to make sure it works for others before I upload it (since the consequences of failure could be bad). If you want to volunteer, I strongly recommend you make a copy of your manifest first, and use the commands individually first (rather than using any scripts) so that you can abort if things go wrong.

avatar
Lostinablock: For anyone having reCAPTCHA errors, it is because of a bug occurring in the detection of it in the script. Some page update broke it. IF when login in through the browser you don't see appear it won't be a problem through gogrepoc.py.
Thanks to MattMills https://github.com/eddie3/gogrepo/issues/65#issuecomment-1250761919 for finding this out.
This affects both eddie3 and
Kalanyr (*the one I'm using) versions.

I've wasted 4 hours trying to extract cookies... what a mess theses are now, _abck ak_bmsc bm_mi ....??
Anyway

Find line:
if len(etree.findall('.//div[@class="g-recaptcha form__recaptcha"]')) > 0:

Replace 0 with 99999999 , save.

Login through browser, if no recaptcha then do the: python gogrepoc.py login
As a warning, that totally disables the recaptcha detection (because the form isn't going to be included a billion times). Will have to dig into it to find out why it's appearing on the standard login form too and what the difference is when it's actually asking for a reCAPTCHA.




ETA - I've identified what's causing the false positive, registration now *always* requires a ReCaptcha and the registration form is included on the login page. If someone has a copy of the login page with the recaptcha showing for logging in , could I get them to save a copy and send it to me please ? I need to ID where in the page the login recaptcha shows up when it's present.
Post edited September 21, 2022 by Kalanyr
I've made a best effort guess at where a reCaptcha for the login form specifically would appear and I've uploaded this version to the dev branch. You will have to login once before doing anything to swap over.

If you have problems do also try logging in via the website, and if you get a recaptcha there, please save the page and contact me.

ETA - Having some issues with renewing the token during *really* longs updates like full updates. Weirdly it doesn't happen during download sessions (probably because each action in a download is relatively fast) as compared to updates which may be waiting a while between individual items to avoid getting rate limited) .I think my implementation is too fragile (I only try to refresh tokens once). So I guess I'm working on it (but this is really annoying to test since it takes an hour). If you're using the dev branch, you can actually ignore it asking you to login in again (in this circumstance), the token gets refreshed fine if you just use -resumemode onlyresume and let it continue.

ETA2 - I really should consider doing updates in parallel, as long as the manifest is shared and I take a write lock before writing to anything external it should work fine. Not to self, have to lock around in manifest and resume manifest together in this case because do not want them to desynch.

ETA3 - I have made the token renewal more robust and it works in testing, so pushed to dev. Can probably improve it further by separating it out a bit and giving it it's own 3 independent 3 retries but it's fixed the original issue, so going t o let it continue to run through the rest of the update session and see how it goes.

ETA4 - Well 2 updates:
1) Unfortunately had to implement the separate retry limit, the extreme case happens more often than I'd hoped, but in hind sight that makes sense, recovering from errors takes much longer than other individual actions, so the chance of being in a retry loop when the token expires is disproportionately high. Going to let that run over night and hope it fixes things.

2) I think GOG doesn't actually validate tokens for expiry during certain exchanges, because it freaks out the first time a token is renewed during a full update but not during downloads (normal udates don't take long enough for tokens to expire except in the most pathological cases), unfortunately I don't know if I have the information to deal with this other than by the brute force method.

ETA5 - Refactoring the token renewal seems to have worked so uploaded that to dev.

ETA6 - Removed the load_cookies call from download, that I no longer remember why it was placed where it was and that is now worse than useless.
Post edited September 23, 2022 by Kalanyr
avatar
Kalanyr: I've made a best effort guess at where a reCaptcha for the login form specifically would appear and I've uploaded this version to the dev branch. You will have to login once before doing anything to swap over.

If you have problems do also try logging in via the website, and if you get a recaptcha there, please save the page and contact me.

ETA - Having some issues with renewing the token during *really* longs updates like full updates. Weirdly it doesn't happen during download sessions (probably because each action in a download is relatively fast) as compared to updates which may be waiting a while between individual items to avoid getting rate limited) .I think my implementation is too fragile (I only try to refresh tokens once). So I guess I'm working on it (but this is really annoying to test since it takes an hour). If you're using the dev branch, you can actually ignore it asking you to login in again (in this circumstance), the token gets refreshed fine if you just use -resumemode onlyresume and let it continue.

ETA2 - I really should consider doing updates in parallel, as long as the manifest is shared and I take a write lock before writing to anything external it should work fine. Not to self, have to lock around in manifest and resume manifest together in this case because do not want them to desynch.

ETA3 - I have made the token renewal more robust and it works in testing, so pushed to dev. Can probably improve it further by separating it out a bit and giving it it's own 3 independent 3 retries but it's fixed the original issue, so going t o let it continue to run through the rest of the update session and see how it goes.

ETA4 - Well 2 updates:
1) Unfortunately had to implement the separate retry limit, the extreme case happens more often than I'd hoped, but in hind sight that makes sense, recovering from errors takes much longer than other individual actions, so the chance of being in a retry loop when the token expires is disproportionately high. Going to let that run over night and hope it fixes things.

2) I think GOG doesn't actually validate tokens for expiry during certain exchanges, because it freaks out the first time a token is renewed during a full update but not during downloads (normal udates don't take long enough for tokens to expire except in the most pathological cases), unfortunately I don't know if I have the information to deal with this other than by the brute force method.

ETA5 - Refactoring the token renewal seems to have worked so uploaded that to dev.

ETA6 - Removed the load_cookies call from download, that I no longer remember why it was placed where it was and that is now worse than useless.
Thanks. Trying to update with ~560 items. Works so far. Glad that at least login is working again.
Can login again with the DEV version, Thank you!
Making a "python gogrepoc.py update -os windows -lang de en -full" now with 595 files.
avatar
Genericjohndoe: Can login again with the DEV version, Thank you!
Making a "python gogrepoc.py update -os windows -lang de en -full" now with 595 files.
From where I can download the dev version?
avatar
Genericjohndoe: Can login again with the DEV version, Thank you!
Making a "python gogrepoc.py update -os windows -lang de en -full" now with 595 files.
avatar
pigulici: From where I can download the dev version?
You can get the dev version here:
https://github.com/Kalanyr/gogrepoc/tree/dev

Will try to update my library in a week or so. So can't comment on it yet, but thanks Kalanyr for updating it!
I've merged the dev branch into the master branch now.

Be aware that you'll have to login once to change over to the new token login system. One of the nice things about the token system though, is that it's auto refreshing, so theoretically you won't get logged out once a year any more.
avatar
Kalanyr: I've merged the dev branch into the master branch now.
Thank you Kalanyr for your continuous work on this project.
Without this tool, managing offline installers would be unfeasible.
I wish Gog supported it like they do for gogdb..
Post edited October 03, 2022 by phaolo
Edit : nope, problems solved.

Edit2: yes, everything works perfectly fine now. Congratulations and thank you.

Edit3: apart from a bunch of "request failed: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out. (1 retries left) -- will retry in 5s..." but I'll look into the meaning of all this later... :-/
Post edited October 03, 2022 by Telika
I installed all that it is in readme, for gogrepoc, still, when I give 'gogrepoc.py login', I got this error:

O:\gog> python gogrepoc.py login
File "O:\gog\gogrepoc.py", line 8
<!DOCTYPE html>
^
SyntaxError: invalid syntax

PS O:\gog> python gogrepoc.py -h
File "O:\gog\gogrepoc.py", line 8
<!DOCTYPE html>
^
SyntaxError: invalid syntax

anyone can help me?
Post edited October 06, 2022 by pigulici
avatar
pigulici: I installed all that it is in readme, for gogrepoc, still, when I give 'gogrepoc.py login', I got this error:

O:\gog> python gogrepoc.py login
File "O:\gog\gogrepoc.py", line 8
<!DOCTYPE html>
^
SyntaxError: invalid syntax

PS O:\gog> python gogrepoc.py -h
File "O:\gog\gogrepoc.py", line 8
<!DOCTYPE html>
^
SyntaxError: invalid syntax

anyone can help me?
Sure you downloaded the raw file ?

From there : https://raw.githubusercontent.com/Kalanyr/gogrepoc/master/gogrepoc.py ?