It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3662 posts)(3662 posts)
(3662 posts)
Pages:
1
50
100
150
195
196
197
198
199
245
This is my favourite topic
Posted February 22, 2022
avatar
Phin77: Edit: I suppose you could simply use -skipfiles patch_*.*. I got too fancy I suppose...
Thanks for the good tip!
This is exactly what I was missing in my gogrepoc commands - ability to exclude patches.
Posted February 22, 2022
avatar
mrkgnao: Given that MaGog collected hundreds of pieces of information on every single one of the thousands of games in GOG's catalogue every six hours for more than six years with rarely a noticeable effect, I consider the stress created by a full update utterly negligible.
MaGoG was a single entity. Heck, even my 10000-30000 requests, if taken by themselves are a drop in the ocean.

Now, empower hundreds, maybe thousands, of users to do the same thing (and they're not all doing it nice and sequential like Magog was over the course of a day, its more random than that with a bunch of live users) and its another story.

This is really a case of "What if everyone was doing as I do?".

I'm pretty sure that between all 3 unofficial clients, we're talking about hundreds of users at this point. If those clients were more mainstream (ie, really well polished extremely easy to setup and use with a gui), we'd be talking thousands.
Post edited February 22, 2022 by Magnitus
Posted February 22, 2022
avatar
mrkgnao: Update is not really the stressing operation; download is.
I doubt blotunga meant just updating the manifest but not downloading the files.. what would that accomplish? O_o

avatar
mrkgnao: I do a regular update (i.e. flagged games) whenever I notice a GOG flag, which could easily be several times a day. I do a full update about once or twice a month.
Wouldn't that just download way more files in those months (4-6 for me), instead of just doing 1 full update?
Expecially for new big games that get frequent changes (if you don't have them, I'm speaking in general).
Posted February 22, 2022
avatar
mrkgnao: Update is not really the stressing operation; download is.
avatar
phaolo: I doubt blotunga meant just updating the manifest but not downloading the files.. what would that accomplish? O_o
The purpose of that is to find all those installer updates that GOG did not flag (and there are a lot of those). You do a full update, then download those unflagged new installers, not your entire library. It's not that different from a regular update, except that it takes much longer and finds unflagged games. But it's essential to do that regularly, otherwise your library is not up to date.

avatar
mrkgnao: I do a regular update (i.e. flagged games) whenever I notice a GOG flag, which could easily be several times a day. I do a full update about once or twice a month.
avatar
phaolo: Wouldn't that just download way more files in those months (4-6 for me), instead of just doing 1 full update?
Expecially for new big games that get frequent changes (if you don't have them, I'm speaking in general).
Yes. It does result in more downloads for those games that are updated several times in that period. But if you don't want to keep your library up to date, that's your choice. What would happen if sometime in those 4-6 months GOG goes away (or the game, or one of its files, is removed from your library)? You'd be left with a non-up-to-date game or file.
Post edited February 22, 2022 by mrkgnao
Posted February 22, 2022
avatar
blotunga: Not everyone has the same workflow. I for example have gogrepo on a Synology NAS, automatically updating my library weekly and full once a month. The first time I downloaded my library it took 5+ days.
avatar
phaolo: Isn't that.. way too frequent? O_o
I recall Kalanyr said to use gogrepo sparingly to not stress Gog's servers (expecially during sales).
I run it like 3 times a year..
Doing a full update once a month doesn't sound that bad...

I also don't normally run an update for months (two or three, sometimes more, months, if I just don't feel like doing it)... but I admit that right now I wish I had run a full update recently, if there really are issues now with gogrepo working. :D

Well, at least I have a "pretty recent" (ie. some months old) snapshot of my games, and at least downloading the games I've bought during the last few months shouldn't be too hard, even manually. So at least it could be worse.

avatar
Magnitus: The thing is that if you wait until you got 200+ games to update, you will help saturate their CDN for several hours while you download everything.
Then again, if you wait longer before the next update, you might many times skip downloading versions that appeared in between. So on the average you will be downloading less from GOG servers, sometimes much less, like when I skipped a couple of Cyberpunk 2077 updates before I re-downloaded the game, I avoided downloading the whole game again a couple of times.
Post edited February 22, 2022 by timppu
Posted February 22, 2022
low rated
I haven't followed this thread in a while.
I have some questions.
Please excuse my rudeness but I don't have time to read the few dozen pages since the last time I checked the thread so I will just stright out ask.

1. What is the current most actively worked on / updated fork ?
2. Does it use GOG API for the *downloads* already ?
3. Does it support download pause / resume (within same chunk / file) ?
4. Would it survive IP change mid download ?
5. Does it allow getting checksums / downloads of non-latest builds ?

I would appreciate the response.
Feel free to make it in-depth.
Posted February 22, 2022
avatar
mrkgnao:
Basically this. Not every update is flagged, if you accidentally click in your library to an updated/new game, gogrepo update will miss it etc. I prefer to play it safe.
avatar
Phin77:
Nice idea with the filters, I only identified a handful (too lazy), but could you paste somewhere your full list, I think I might add some more :)
I'm using these at the moment: "patch_*.exe" "patch_*.bin" "patch_*sh" "*_(32bit)_*.exe" "*_(32bit)_*.bin"
Post edited February 22, 2022 by blotunga
Posted February 22, 2022
avatar
timppu: Then again, if you wait longer before the next update, you might many times skip downloading versions that appeared in between. So on the average you will be downloading less from GOG servers, sometimes much less, like when I skipped a couple of Cyberpunk 2077 updates before I re-downloaded the game, I avoided downloading the whole game again a couple of times.
Fair enough.
Posted February 22, 2022
avatar
blotunga: Basically this. Not every update is flagged, if you accidentally click in your library to an updated/new game, gogrepo update will miss it etc. I prefer to play it safe.
That's why I don't even bother with the non-full update ¯\_(O_o)_/¯

avatar
blotunga: could you paste somewhere your full list, I think I might add some more :)
For installers I use:
(I've manually downloaded the few required found by the community)

-skipfiles patch_* setup_*_32_* setup_*_(32bit)*

For extras the list is long tho (I wish it allowed regex):

-skipfiles *.dmg *.sh *.gz *_german.zip *_german_2* *_german_v* *_de.zip *_french.zip *_fr.zip *_spanish.zip *_es.zip *_italian.zip *_ita.zip *_it.zip *_russian.zip *_russian_2* *_russian_v* *_ru.zip *_polish.zip *_pl.zip *_zh.zip *_handbuch.zip *_manuel.zip *_avatar*.zip *_trailers.zip *_sample.zip *_demo.zip *_wav.zip *_wma.zip blitzkrieg_v2*.zip driftmoon_vanilla* setup_outcast_* rott_2013_level* setup_strike_suit* bonus_pack_*_.zip *_redkit.zip all_goodies.zip

I skip some of these because they're old\alt versions, or they're mostly duplicates, or they're way too big (rott editor), etc.
Also, some of the similar entries are needed due to Gog's.. variable naming convention.
Post edited February 22, 2022 by phaolo
Posted February 22, 2022
I'm new to the party. I just downloaded gogrepo.py from https://github.com/eddie3/gogrepo and ran it on my Linux machine. 1) login was successful (to my surprise, I was afraid of the reCAPCHA thing). 2) downloading the game details only gets me "xml parsing error trying to get md5 data for ..."

I looked in this discussion for fixes, but I'm not sure what to change in the script. Even though it says it's not for Python 3 yet (README.md from the github repo), it actually runs with python 3.9.9 on my machine. Is the md5 error related somehow?

I cannot find a line with
response.url.replace('?', '.xml?')
as mentioned in this thread (somewhere), only
page.geturl().replace('?', '.xml?')
so I don't know how to fix this.

Help appreciated.
Post edited February 22, 2022 by Andtha
Posted February 22, 2022
Since no one has replied yet, I say what I can:

avatar
B1tF1ghter: 1. What is the current most actively worked on / updated fork ?
2. Does it use GOG API for the *downloads* already ?
3. Does it support download pause / resume (within same chunk / file) ?
4. Would it survive IP change mid download ?
5. Does it allow getting checksums / downloads of non-latest builds ?
1. Kalanyr's fork is at least the one most talked about in this thread. No idea if there is some other "unknown" fork that is currently more actively updated.

2. No idea.

3. You can exit the script (while it downloads) anytime you want with Ctrl-C, and then continue by running the same gogrepo download command again. It will resume the downloads from where you left (I think also partial downloads).

Just remember that if you wait so long (before resuming your downloads) that some of the files have been updated/replaced on the GOG servers, then you have to run also "gogrepo update" again, to get the details for the changed files.

4. Yes, if you "pause" and "resume" the download as mentioned in the previous point.

5. I don't think so.
Posted February 22, 2022
avatar
Andtha: I'm new to the party. I just downloaded gogrepo.py from https://github.com/eddie3/gogrepo and ran it on my Linux machine. 1) login was successful (to my surprise, I was afraid of the reCAPCHA thing). 2) downloading the game details only gets me "xml parsing error trying to get md5 data for ..."

I looked in this discussion for fixes, but I'm not sure what to change in the script. Even though it says it's not for Python 3 yet (README.md from the github repo), it actually runs with python 3.9.9 on my machine. Is the md5 error related somehow?

I cannot find a line with
response.url.replace('?', '.xml?')
as mentioned in this thread (somewhere), only
page.geturl().replace('?', '.xml?')
so I don't know how to fix this.

Help appreciated.
That particular fix is indeed similar for the line that you identified:
https://github.com/eddie3/gogrepo/blob/2cd3ac4a77d3e5104662fcfb4d96e7129a758bfe/gogrepo.py#L360

You change it to:
tmp_md5_url = page.geturl() + '.xml'

(Keep the indentation of the line identical).

It won't resolve all of the new issues (bad xml responses, gzipped responses) but it will resolve trying to download with a bad url.
Posted February 22, 2022
avatar
PraetorianWolfie: You change it to:
tmp_md5_url = page.geturl() + '.xml'
That worked! Thank you very much!
Posted February 22, 2022
low rated
avatar
timppu: Since no one has replied yet, I say what I can:
I was about to ask why I got downvoted to hell for asking legit questions.
I really don't get it.
My questions are totally legit.
I dare to say, even, crucial.

Thank you for your answer.

avatar
timppu: 1. Kalanyr's fork is at least the one most talked about in this thread. No idea if there is some other "unknown" fork that is currently more actively updated.
There's like a metaphorical billion of forks.
https://github.com/eddie3/gogrepo/network/members
The problem is 99% of them are under-documented, and they don't state what they changed, also some of them are incorrectly * merging pull requests out of different projects.
There's no changelog for 99% of these forks either.

* You shouldn't exactly be doing that. I mean. You "CAN". But you shouldn't. It creates confusion and can lead to unexpected issues.

I tested gogrepopy in july/august 2021 for the first time.
I used vanilla eddie3 build.
According to github it's not been commited to for 3 YEARS.
Suprisingly, there are open pull requests dated even in 2021 (that are... getting merged in random forks in random order...).

There's the Kalanyr's fork.
No "changed from eddie3 faq" that I could find.

Then there's this:
https://github.com/Magnitus-/gogrepoc

And then there's this clusterf**k:
https://github.com/eddie3/gogrepo/network
( GO FIGURE )
One would think this question alone would make my post worthy of a response instead of being utterly ignored but I guess I am expecting too much from this forum.

The fact that 99% (100% ? I don't have time to check) of these forks directly copy paste the FAQ from the master build doesn't help.


My questions 2, 3, and 4 came because I had an issue before and had the same-ish issue now when I tried to download things from GOG.
Since I am an IT versed in few related areas I debugged this a bit and quickly found out that GOG's server backend is configured in a totally backwards ancient way that resembles *decisions* from before 2000.
TLDR:
403 fest on *ANY* issue. A N Y issue.

As I said, I used the vanilla eddie3 half a year ago and at that point it was so broken for me that I was only able to use it for checksums.
Basically at any pocket loss or any random network hiccup the tool refused to resume downloads of the same FILES and the entire file that was mid-download had to be downloaded from the start.
That forced me to just gut the checksums with the tool and download the files with browser (which somehow happaned to be more reliable then).

In recent days I tried to download some other game, *with browser*, this time tho I am on a metered mobile broadband connection, which is unreliable, can have a random reconnect thus IP change.
Here's where the fun begins.
From my testing it seems that:
Your IP changes mid download - download DEAD - refuses to reconnect. Then sudden 403.
You try to download more than 1 file at a time (NO MATTER HOW SMALL) - first you get a sizeable delay - as a bonus you get added instability (on GOG's side).
You pause 1st download. Start 2nd. Resume 1st - you ISNTANTLY get a network error, and then you get 403 shoved into your face on your 1st download.
You try to download the same file twice at the same time (for whatever reason) - 403.
Also, downloads tend to suspiciously die at 99% even when my network connection, network equipment, and entire system is fine.
To say that all this pissed me off would be an understatement.

Now, GOG's " "support" " pages have this:
https://support.gog.com/hc/en-us/articles/212807525--Error-403-when-downloading-with-a-browser?product=gog
that basically says "use our 'TOTALLY OPTIONAL' client".
No thanks. It's attrocious.
But this says 2 things:
1.Gog doesn't care to fix their backend **
2.Galaxy API supports resume

Hence my 2,3 and 4 questions.

** That's the thing.
Any SANELY configured file server survives both IP change and network issues (even if the download requires login).
There's A LOT (and I mean LUDICROUS amount) of file servers in the field that are like that. Perhaps even most of them out there are like that.

Not GOG's tho.
What's even the point of using a token in the url if the server throws a tantrum on a random issue...
Sigh...
Posted February 22, 2022
avatar
trigonometrie: Id welcome a common download api.
The valve steam launcher is a giant behemoth and every other launcher also brings its own copy of chromium embedded framework or electron. This is duplication that isn't necessary.
garbage chromium broaf
they are so slow all the time... cant even get proper small pics and takes time to load them when you scroll..
Pages:
1
50
100
150
195
196
197
198
199
245
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3662 posts)(3662 posts)
(3662 posts)