It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3849 posts)(3849 posts)
(3849 posts)
Pages:
1
50
100
150
198
199
200
201
202
250
257
This is my favourite topic
Posted February 24, 2022
avatar
blotunga: Nice idea with the filters, I only identified a handful (too lazy), but could you paste somewhere your full list, I think I might add some more :)
I'm using these at the moment: "patch_*.exe" "patch_*.bin" "patch_*sh" "*_(32bit)_*.exe" "*_(32bit)_*.bin"
avatar
Phin77: Thanks much!

Mine mostly matches the one posted above but I'll post it here minus the line breaks. (I hated the way my last post looked):
For exclusion lists like those, it would nice to get a clear log of all files that the script decided to skip. So that you can check it afterwards, in case there were files that you DO want to download after all.

(If that info already goes to the gogrepo log file, I guess it is simple to grep the relevant lines from there afterwards...)

I guess there should be also an exlusion list to that exclusion list, so that "these particular files will be excluded from the exclusion". :)
Post edited February 24, 2022 by timppu
Posted February 24, 2022
avatar
lupineshadow: Really phaolo? No idea why people are downvoting this unpleasant person?
I wouldn't blame bots...
Well, he was behaving kinda normally before that reply to Timboli O_o'
Posted February 24, 2022
low rated
avatar
lupineshadow: Really phaolo? No idea why people are downvoting this unpleasant person?
I wouldn't blame bots...
avatar
phaolo: Well, he was behaving kinda normally before that reply to Timboli O_o'
that was just a normal advice, people should appreciate advices instead of taking them as personal attacks
Posted February 24, 2022
avatar
timppu: For exclusion lists like those, it would nice to get a clear log of all files that the script decided to skip. So that you can check it afterwards, in case there were files that you DO want to download after all.
(If that info already goes to the gogrepo log file, I guess it is simple to grep the relevant lines from there afterwards...)
Yeah, you can see it in the log, like:
skip patch_gamename.exe (matches "patch_*")

avatar
phaolo: Well, he was behaving kinda normally before that reply to Timboli O_o'
avatar
Orkhepaj: that was just a normal advice, people should appreciate advices instead of taking them as personal attacks
Not exactly, his tone was really wrong.
A bit too entitled and aggressive, expecially for a free software.
He could have expressed the same points with politeness and Timboli's reply would have been different.
Post edited February 24, 2022 by phaolo
Posted February 24, 2022
low rated
avatar
phaolo: Well, he was behaving kinda normally before that reply to Timboli O_o'
avatar
Orkhepaj: that was just a normal advice, people should appreciate advices instead of taking them as personal attacks
EXACTLY.
Especially if the advice is revolving around code, even more so if the code in question is being used by many individuals.

--------------------

The premise of "you can't criticise ( / give non-neutral feedback to ) something because it's FREE" (insert "FREE?!" meme here) is down right satirical level of hilarious.
Some people just can't take criticism (or any feedback whatsoever in some cases) regardless of the tone, I have met such people both online and in person.
Many even use it as an excuse to COMPLETELY ignore the actual *meaning* of the message delivered to them.

--------------------

I took a glimpse at some sections of the code (kalanyr's fork, dev branch) and the lang section notably has incomplete list of GOG-originated languages.
I noticed lack of ukrainian (perhaps some others are missing too, I don't know).
It's unclear at this point if separate installer builds exist with files different for that language.
I noticed (long time ago) Metro had this in the downloads
https://www.gogdb.org/product/1436434037#downloads
https://www.gogdb.org/product/1430740172#downloads
Some people prefer that dub, but then again FOR NOW I am uncertain if actual builds with separate installers for that language having separate files (only present in those isntallers) exist.

--------------------

Btw, it would certainly be nice to be able to get manifests for older builds of games.
I would very much appreciate that for a number of reasons (imo GOG should always provide both the latest version, the vanilla RTM version and ALL of the patches, nagging support or using Galaxy should not be neccessary to get the RTM version).

--------------------

Full manifest update with the mrkgnao fixes worked.
But I have some observations.
First of all, to reinstate what I said earlier:
WITHOUT the fixes the script not only hangs but *NETWORK hogs* - seemingly downloads infinite amounts of data (I didn't have packet sniffer ready so I don't know what exactly it downloads indefinitely).

WITH fixes:
*when my speed is below 0.5 Mbps I get the timeouts*
When I made my network stay above that speed there were definitely spikes above 0.5 Mbps during manifest update (I monitored with nload).

In case this is somehow package version related, I run this on:
python 3.10.2-1
python-html5lib 1.1-11
python-pyopenssl 21.0.0-5
python-html2text 2020.1.16-5
python-pydbus 0.6.0-9

BEWARE, these are Arch packages, and Arch *frequently* applies custom patches.
Contrary to popular belief Arch packages often are NOT 100% vanilla due to that.

--------------------

I have done resume testing on kalanyr's (DEV BRANCH) fork with mrkgnao fixes.
Results:
- CTRL C, then manual resume [PASS]
- IP change mid download w/o CTRL C, auto-resume [PASS]
- hard disconnect * during download w/o CTRL C, IP change, auto-resume [PASS]
- hard disconnect during download w/o CTRL C, no IP change, auto-resume [PASS]
- CTRL C, no LAN dropout, manual resume on different IP [PASS]

* meaning air gapping entire host system by eth cable unplug

Full disclosure:
I have made all *resume* testing on same file *download*.
( I triggered a bunch of intentional failure points in succession during the same file *download* ).
It may or may not have impacted the results.

--------------------

I have already successfully downloaded a bunch of 4 GB chunks and it generally seems like this will now work for me.
So, essentially, kalanyr's fork along with mrkgnao fixed most of my issues.

avatar
mrkgnao: ...
avatar
Kalanyr: ...
You guys saved my bacon.
I very much appreciate and thank you for that :)

--------------------

Now, for the issue I hit consistently and it seems to reproduce to the same result 100% of the time.

issue description
https://www.toptal.com/developers/hastebin/nezeqisame.sql
log
https://www.toptal.com/developers/hastebin/oxogicovip.sql

I dunno if it will help fix something or not but here goes nothing.
( the related product is a demo that is still obtainable so I guess anyone could test it on their config )
Post edited February 24, 2022 by B1tF1ghter
Posted February 24, 2022
avatar
timppu: For exclusion lists like those, it would nice to get a clear log of all files that the script decided to skip. So that you can check it afterwards, in case there were files that you DO want to download after all.
(If that info already goes to the gogrepo log file, I guess it is simple to grep the relevant lines from there afterwards...)
avatar
phaolo: Yeah, you can see it in the log, like:
skip patch_gamename.exe (matches "patch_*")
Good. I guess those lines can be grepped somehow, not sure if there are other non-relevant "skip"-lines in the log as well, but at least something like |grep skip |grep matches should probably work.

How about gogrepoc clean, does it support such exclusion lists too so that you can get rid of those same files if you have them already downloaded? Or do you have to remove them manually first?

I presume if you just run normal clean, then it will consider those extra files as relevant, as they are mentioned in the original manifest file?
Posted February 24, 2022
avatar
timppu: Good. I guess those lines can be grepped somehow, not sure if there are other non-relevant "skip"-lines in the log as well
Some things can be avoided with other parameters like: "-os windows -lang en -skiphidden -skipextras"
I don't think those appear at all in the log (but it's good IMO).

avatar
timppu: How about gogrepoc clean, does it support such exclusion lists too so that you can get rid of those same files if you have them already downloaded? Or do you have to remove them manually first?
Mm.. I'm not sure, I've only used "skipfiles" with the "download" parameter.
Posted February 24, 2022
avatar
MatrixRaven: Excellent, thanks. I've downloaded it and will give it a try. I like your apps. Thanks for sharing them.
No worries, thank you for your kind words, and hope my program(s) serves your need. :)
Posted February 24, 2022
avatar
Lhun Duum: So I tried a full update using a VPN to connect from a complete different continent and this time around instead of dozens of games not updating correctly, the only game with which I had an issue was ironically The Witcher 3. So I think there might be some truth to my hypothesis of some servers having problems in some regions.
That is interesting info, thanks. Makes sense given that they are using a CDN and you might not be pinging the exact same copy of the content depending on where you access the data from. If the content is malformed in some way, it could be in select locations only.
Post edited February 24, 2022 by Magnitus
Posted February 24, 2022
avatar
B1tF1ghter: The level of hypocrisy you show here is hilarious.
You are a laugh a minute, and I see no hypocrisy on my side of things, it only exists in your deluded mind.

You come here as a lunatic, who is a busy man, but not too busy to cast aspersions, attack others, and just be downright judgmental and rude with your incredibly messy posts and lack of insight.

You clearly think your own way is the only way.

As for MD5, the only time I bother with them myself, is when what I am downloading is going to be archived. Most programs I download and use, I never bother checking them against any provided MD5, and most of the time there isn't any MD5, so clearly many folk don't bother. You bother when it is important enough, has always been my mantra.

Many folk in this thread, freely give of their time and knowledge and seek to be helpful to others.

So please stop hijacking this thread with your nonsense and personal vendetta and angst and skewed view of everything.
Posted February 24, 2022
avatar
Lhun Duum: So I tried a full update using a VPN to connect from a complete different continent and this time around instead of dozens of games not updating correctly, the only game with which I had an issue was ironically The Witcher 3. So I think there might be some truth to my hypothesis of some servers having problems in some regions.
avatar
Magnitus: That is interesting info, thanks. Makes sense given that they are using a CDN and you might not be pinging the exact same copy of the content depending on where you access the data from. If the content is malformed in some way, it could be in select locations only.
GOG sometimes has synchronization issues where new data isn't actually properly copied to all the servers. That's gonna make this fun to test for me, since I suspect I'm on a different continent to most of you.
Posted February 24, 2022
avatar
Kalanyr: GOG sometimes has synchronization issues where new data isn't actually properly copied to all the servers. That's gonna make this fun to test for me, since I suspect I'm on a different continent to most of you.
I think an interesting solution (although one that costs a bit of money) would be something similar to what I originally wanted to implement to speed up Timboli's download speed: You setup an intermediate service in the cloud (in a region of your choosing) that talks to gog and you do your downloads via that service.

I think digital ocean is a promising candidate as their egress costs seem to be a lot lower than a lot of their competitors (0.01$ per outbound gb vs 0.09$ per outbound gb for aws).

With the devops culture nowadays, you could setup all the infra via api calls without a need for manual user-end intervention although they'd need to remember to tear it down not to incur an ongoing bill.

Anyways, I'll investigate later on whether this would resolve an actual long-term problem that won't go away with the gog cdn.
Post edited February 24, 2022 by Magnitus
Posted February 24, 2022
Concerning all the posts on prior pages about updating/downloading and mentions of results of game updates being different with VPN...

I've noticed lately that my speeds (whether a sale is ongoing or not) is impacted a great deal depending on 1) the game downloading or 2) amount I'm downloading. I'm on a 1Gbps connection behind a 10Gbps pipe. I can start downloading something sizeable, say Cyberpunk 2077 or 70GB of updates across 3-4 games, and initially speeds are 6-8MB/s per file and maybe an hour later all four files being downloaded are 0.01MB/s. Soooo....yeah. That's a thing. I suspect it's the CDN but maybe somebody with more knowledge can illuminate upon this?
Posted February 25, 2022
patch_the_riftbreaker_1.41_(51250)_to_1.42_(51255).exe
This file downloaded but didn't import.

Ran a new update specific to this game. Log file is too long to post.

During handling of the above exception, another exception occurred:
This message appears 15 times.

The following is the 15th occurrence.

Traceback (most recent call last):
File "gogrepo.py", line 713, in filter_downloads
fetch_file_info(d, True,updateSession)
File "gogrepo.py", line 669, in fetch_file_info
md5_response = request(updateSession,tmp_md5_url)
File "gogrepo.py", line 243, in request
return request(session=session,url=url, args=args, byte_range=byte_range, retries=retries-1, delay=HTTP_RETRY_DELAY)
File "gogrepo.py", line 243, in request
return request(session=session,url=url, args=args, byte_range=byte_range, retries=retries-1, delay=HTTP_RETRY_DELAY)
File "gogrepo.py", line 243, in request
return request(session=session,url=url, args=args, byte_range=byte_range, retries=retries-1, delay=HTTP_RETRY_DELAY)
File "gogrepo.py", line 229, in request
response = session.get(url, params=args,stream=stream,timeout=HTTP_TIMEOUT)
File "C:\Users\<User>\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 525, in get
return self.request('GET', url, **kwargs)
File "C:\Users\<User>\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 512, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\<User>\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 662, in send
r.content
File "C:\Users\<User>\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\models.py", line 827, in content
self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
File "C:\Users\<User>\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\models.py", line 756, in generate
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out.
20:10:25 | failed to fetch https://www.gog.com/downloads/the_riftbreaker/en1patch1 because of non-HTTP Error
20:10:28 | saving manifest...
20:10:29 | saved manifest
20:10:29 | saving resume manifest...
20:10:29 | saved resume manifest
20:10:29 | --
20:10:29 | total time: 0:02:49.525497
Posted February 25, 2022
in case there were files that you DO want to download after all.
Pages:
1
50
100
150
198
199
200
201
202
250
257
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3849 posts)(3849 posts)
(3849 posts)