It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
avatar
seattleandrew: I'm interested!
Here you go : https://1drv.ms/u/s!AkusSFHlKO4Lni6Sg3YRlRzvNOCY?e=gR8uxN

Edit the variables at the beginning of the script to tailor the downloads to your needs and to set paths.

Let me know if you have any issues, comments or suggestions.
avatar
Kalanyr: How many people are still using Python 2 and wouldn't be able to change to Python 3 for some reason ?
avatar
teceem: I'd like a Windows GUI and not have to use any form of command line interface at all. But that's just me.
I guess it is better any GUIs are made as separate frontends. I think the point of the base gogrepo.py is to be cross-platform, ie. it works on most devices and OSes that can run python scripts.

I am personally using gogrepo in Windows, Linux Mint and Raspbian (Raspberry Pi 4). More and more only on Raspbian (e.g. right now I took a ssh connection from this Windows 7 PC to my Raspberry Pi, and started "python3 gogrepoc.py update -lang en -os windows").
Post edited February 15, 2020 by timppu
is it possible to set custom paths for gogrepo's cookie file (or whatever it uses to keep authorized into your account) and database?

Coz I remember it being a thing for games themself, but thats about it
Here is a quick question: I downloaded many games from my library, but it seems some Linux installers were missing (seems like I botched the command line arguments).

I now wish to get only the files I'm missing (including the games I added to my library after that moment). Also, I'm not sure the updated flag that GOG places on my library is intact (just looking at the log clears it -- not good!).

As I understand it, there is no option to skip the files already present on disk (usually called "no-clobber"), or to check the hash and skip the file if OK. Am I missing something?
avatar
Gede: Here is a quick question: I downloaded many games from my library, but it seems some Linux installers were missing (seems like I botched the command line arguments).

I now wish to get only the files I'm missing (including the games I added to my library after that moment). Also, I'm not sure the updated flag that GOG places on my library is intact (just looking at the log clears it -- not good!).

As I understand it, there is no option to skip the files already present on disk (usually called "no-clobber"), or to check the hash and skip the file if OK. Am I missing something?
Gogrepo (both the original and gogrepoc) automatically skip downloading already present files (gogrepoc is a bit more aggressive if the existing file is obviously wrong (eg wrong size)), so you can safely do a full manifest update and then use download to get anything missing.
avatar
Gekko_Dekko: is it possible to set custom paths for gogrepo's cookie file (or whatever it uses to keep authorized into your account) and database?

Coz I remember it being a thing for games themself, but thats about it
No, they both go into the same folder as the script. I assume you're looking at a multi-user scenario and also doing something like dividing the manifest up by OS ?
Post edited February 21, 2020 by Kalanyr
I recall there was some issue with the gogrepo file preallocation in Linux, what was the issue again?

So far I've been mostly running gogrepoc in Windows, but I tried to run it on Linux now (both on my Mint PC and Raspberry Pi). The important point is that the USB HDD to which the installers are downloaded is NTFS, as the games will be installed in Windows anyway. Naturally I have installed ntfs-3g in Linux so that it can read and write to NTFS partitions just fine.

python3 gogrepoc,py update & clean seem to work just fine, but when gogrepoc tries to download, the "preallocation" takes several minutes, and after that I start getting odd errors. The screen capture is at the end of this messge.

Also, I just tested by redirecting the downloads to an ext4 Linux-partition instead... and sure enough, the downloads start just fine and the preallocations also take mere seconds, instead of several minutes. (I recalled using gogrepo earlier in Linux just fine, so I gather either I was directing the downloads to a Linux partition, or the gogrepo version was so old that it didn't yet have the preallocation feature?)

If it is so that the preallocation feature is incompatible with NTFS in Linux (ntfs-3g), is it possible to disable the file preallocation in the script easily, or is it so integrated to everything that it is hard to get rid of? I was never a big fan of that preallocation anyway, seems kinda pointless if you ask me (fragmentation schragmentation, who cares, these are big files).

I guess the other option is that I reformat the USB HDD where I keep my GOG installers to ext4 or xfs and then use some driver in Windows so that Windows can read ext4/xfs partitions, but for simplicity's sake, I'd rather keep it NTFS.

This is what it looks like when running python3 gogrepoc.py download to a NTFS partition (notice the timestamps how long each preallocation takes):

21:18:36 | ------------------------------------------------------------
21:18:36 | attempting preallocating '109576656' bytes for '/media/user/GOG/GOG/!downloading/80_days/setup_80_days_1.17.6_(35219).exe' using posix_fallocate
21:18:42 | attempting preallocating '4293628414' bytes for '/media/user/GOG/GOG/!downloading/a_plague_tale_innocence/setup_a_plague_tale_innocence_1.07_(64bit)_(34188)-1 .bin' using posix_fallocate
21:23:30 | attempting preallocating '4294967294' bytes for '/media/user/GOG/GOG/!downloading/a_plague_tale_innocence/setup_a_plague_tale_innocence_1.07_(64bit)_(34188)-2 .bin' using posix_fallocate
21:28:21 |
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 302, in _error_catcher
yield
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 384, in read
data = self._fp.read(amt)
File "/usr/lib/python3.6/http/client.py", line 459, in read
n = self.readinto(b)
File "/usr/lib/python3.6/http/client.py", line 503, in readinto
n = self.fp.readinto(b)
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.6/ssl.py", line 1012, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.6/ssl.py", line 874, in read
return self._sslobj.read(len, buffer)
File "/usr/lib/python3.6/ssl.py", line 631, in read
v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer
Post edited February 21, 2020 by timppu
avatar
Kalanyr: Gogrepo (both the original and gogrepoc) automatically skip downloading already present files (gogrepoc is a bit more aggressive if the existing file is obviously wrong (eg wrong size)), so you can safely do a full manifest update and then use download to get anything missing.
Ah, excellent! From the dryrun it seemed like I would have to download everything all over again!
avatar
timppu: So far I've been mostly running gogrepoc in Windows, but I tried to run it on Linux now (both on my Mint PC and Raspberry Pi). The important point is that the USB HDD to which the installers are downloaded is NTFS, as the games will be installed in Windows anyway. Naturally I have installed ntfs-3g in Linux so that it can read and write to NTFS partitions just fine.
I would suppose that the problem lies in the filesystem, as you figured out already. Perhaps the fallocate system call for the NTFS filesystem implementation is not optimal, which is likely since NTFS's inner workings are kept pretty secret by Microsoft. Perhaps NTFS does not even support this system call natively, and it is implemented only to satisfy the Linux filesystem API and is not invoked often. And could your HDD be so fragmented that it needs to shuffle things around to satisfy the preallocation? (I think NTFS avoids fragmentation to a point but it is not impossible.)
I actually avoid using NTFS, so I'm afraid I cannot help you there.
Funny that the problem is that gogrepo fails by losing the connection. It took to long?

Now, what are your options? Saving the files locally and then moving them to the HDD would the the simplest option, and it would also cut down your file fragmentation (yeah, I too don't think it matters that much on files you don't use often).

Option B is to modify the source code for gogrepo so that it skips the preallocation step (sadly, there is no switch for it).
Based on my superficial analysis, if you remove (or comment, by adding a "#" at the start of each line) from line 1619 up to (and including) 1706, gogrepo should no longer try to preallocate your downloads.
avatar
timppu: So far I've been mostly running gogrepoc in Windows, but I tried to run it on Linux now (both on my Mint PC and Raspberry Pi). The important point is that the USB HDD to which the installers are downloaded is NTFS, as the games will be installed in Windows anyway. Naturally I have installed ntfs-3g in Linux so that it can read and write to NTFS partitions just fine.
avatar
Gede: I would suppose that the problem lies in the filesystem, as you figured out already. Perhaps the fallocate system call for the NTFS filesystem implementation is not optimal, which is likely since NTFS's inner workings are kept pretty secret by Microsoft. Perhaps NTFS does not even support this system call natively, and it is implemented only to satisfy the Linux filesystem API and is not invoked often. And could your HDD be so fragmented that it needs to shuffle things around to satisfy the preallocation? (I think NTFS avoids fragmentation to a point but it is not impossible.)
I actually avoid using NTFS, so I'm afraid I cannot help you there.
Funny that the problem is that gogrepo fails by losing the connection. It took to long?

Now, what are your options? Saving the files locally and then moving them to the HDD would the the simplest option, and it would also cut down your file fragmentation (yeah, I too don't think it matters that much on files you don't use often).

Option B is to modify the source code for gogrepo so that it skips the preallocation step (sadly, there is no switch for it).
Based on my superficial analysis, if you remove (or comment, by adding a "#" at the start of each line) from line 1619 up to (and including) 1706, gogrepo should no longer try to preallocate your downloads.
I never tested NTFS on Linux (as non-read only mode still came with warnings when I was writing that part), I was assuming exFAT for a shared drive.

If you want to a shared drive between Linux/Mac/Windows I believe the best bet is exFAT with 1024K allocation size (preferably formatted on Windows).

However I don't think that's the issue here.
avatar
Gede: And could your HDD be so fragmented that it needs to shuffle things around to satisfy the preallocation? (I think NTFS avoids fragmentation to a point but it is not impossible.)
I actually avoid using NTFS, so I'm afraid I cannot help you there.
I don't believe it is about the HDD being too fragmented, gogrepo on it works just fine also on Windows and this stuff doesn't get so fast fragmented anyway (rarely changed files).

The reason I use NTFS on the USB HDD where I keep my GOG installers is naturally so that I can easily install the games on my Windows PCs, even though I may do the actual downloading of the installers on Linux (e.g. the low-power Raspberry Pi 4 which is perfect for simple tasks like this which may take a long time to perform).

avatar
Gede: Now, what are your options? Saving the files locally and then moving them to the HDD would the the simplest option,
and it would also cut down your file fragmentation (yeah, I too don't think it matters that much on files you don't use often).
I am unsure how that would work? With gogrepoc clean and update, you point it to the path where your existing installers are (in this case my 5TB USB HDD (NTFS). If you point it to somewhere else, then gogrepoc would just think I have no installers yet downloaded and would download my whole library all over again (which is what it did when I tested it yesterday by directing the downloads to an ext4 Linux partition).

If I can't get this issue solved, then I'd think my only feasible options would be:

1. Reformat that 5TB USB HDD to ext4 or xfs, and then install some ext4/xfs driver on all my Windows PCs so that they can read that filesystem when I connect the USB HDD to them.

2. Not running gogrepo on Linux but keep doing it on Windows only (meh, this was one of the reasons I wanted to buy a low-power Raspberry Pi, it is great for tasks like this where I might leave a computer running for the night or even several days).

avatar
Gede: Option B is to modify the source code for gogrepo so that it skips the preallocation step (sadly, there is no switch for it).
Based on my superficial analysis, if you remove (or comment, by adding a "#" at the start of each line) from line 1619 up to (and including) 1706, gogrepo should no longer try to preallocate your downloads.
Yeah this was what I was interested in, can you disable the preallocation (at least on Linux) by e.g. simply commenting out certain lines in the script, or is the preallocation now so integrated to the whole download function that many parts of the script would have to be totally rewritten?

Maybe I check and try what you suggest, maybe it is really that simple then. Thanks.
Post edited February 22, 2020 by timppu
avatar
Kalanyr: I never tested NTFS on Linux (as non-read only mode still came with warnings when I was writing that part), I was assuming exFAT for a shared drive.

If you want to a shared drive between Linux/Mac/Windows I believe the best bet is exFAT with 1024K allocation size (preferably formatted on Windows).

However I don't think that's the issue here.
E,g, the gogrepoc clean seemed to work just fine on the NTFS drive, it moved obsolete files into !orphaned directory normally. And generally I don't recall ever having any real issues operating NTFS partitions with ntfs-3g, both reading and writing to them (naturally they don't have the Linux file permission capablities etc. but I guess that is beside the point in this case).

Also I have a recollection that I would have run gogrepo a long time ago on Linux, similarly using a NTFS formatted USB HDD... so I figure maybe that was back when it didn't yet have the preallocation? Or then I remember wrong...

So my main question is, is it possible to disable the whole preallocation (like Gede suggested by commenting out certain lines), or does it take more than that?

I guess then my only option is to reformat the USB HDD into ext4 or xfs and find a way to read those filesystems in Windows (should be possible I guess, with some extra driver).
Post edited February 22, 2020 by timppu
avatar
timppu: I recall there was some issue with the gogrepo file preallocation in Linux, what was the issue again?

So far I've been mostly running gogrepoc in Windows, but I tried to run it on Linux now (both on my Mint PC and Raspberry Pi). The important point is that the USB HDD to which the installers are downloaded is NTFS, as the games will be installed in Windows anyway. Naturally I have installed ntfs-3g in Linux so that it can read and write to NTFS partitions just fine.

python3 gogrepoc,py update & clean seem to work just fine, but when gogrepoc tries to download, the "preallocation" takes several minutes, and after that I start getting odd errors. The screen capture is at the end of this messge.

Also, I just tested by redirecting the downloads to an ext4 Linux-partition instead... and sure enough, the downloads start just fine and the preallocations also take mere seconds, instead of several minutes. (I recalled using gogrepo earlier in Linux just fine, so I gather either I was directing the downloads to a Linux partition, or the gogrepo version was so old that it didn't yet have the preallocation feature?)

If it is so that the preallocation feature is incompatible with NTFS in Linux (ntfs-3g), is it possible to disable the file preallocation in the script easily, or is it so integrated to everything that it is hard to get rid of? I was never a big fan of that preallocation anyway, seems kinda pointless if you ask me (fragmentation schragmentation, who cares, these are big files).

I guess the other option is that I reformat the USB HDD where I keep my GOG installers to ext4 or xfs and then use some driver in Windows so that Windows can read ext4/xfs partitions, but for simplicity's sake, I'd rather keep it NTFS.

This is what it looks like when running python3 gogrepoc.py download to a NTFS partition (notice the timestamps how long each preallocation takes):

21:18:36 | ------------------------------------------------------------
21:18:36 | attempting preallocating '109576656' bytes for '/media/user/GOG/GOG/!downloading/80_days/setup_80_days_1.17.6_(35219).exe' using posix_fallocate
21:18:42 | attempting preallocating '4293628414' bytes for '/media/user/GOG/GOG/!downloading/a_plague_tale_innocence/setup_a_plague_tale_innocence_1.07_(64bit)_(34188)-1 .bin' using posix_fallocate
21:23:30 | attempting preallocating '4294967294' bytes for '/media/user/GOG/GOG/!downloading/a_plague_tale_innocence/setup_a_plague_tale_innocence_1.07_(64bit)_(34188)-2 .bin' using posix_fallocate
21:28:21 |
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 302, in _error_catcher
yield
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 384, in read
data = self._fp.read(amt)
File "/usr/lib/python3.6/http/client.py", line 459, in read
n = self.readinto(b)
File "/usr/lib/python3.6/http/client.py", line 503, in readinto
n = self.fp.readinto(b)
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.6/ssl.py", line 1012, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.6/ssl.py", line 874, in read
return self._sslobj.read(len, buffer)
File "/usr/lib/python3.6/ssl.py", line 631, in read
v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer
It's normal for preallocation to take much longer on exFAT/NTFS systems as they really do the pre-allocation (so they are writing 4 GB of data here) , ext3/4 don't actually have the capacity to do true pre-allocation (and it's not really necessary because they are much less subject to fragmentation). Best cross system compatibility in my experience is exFAT with 1024K allocation size formatted on Windows.


Those errors don't look related to the file system, they are SSL connection errors. They also aren't related explicitly to a gogrepoc line, so it's something happening to an existing connection that's handling things on its own.

I wonder if it's taking so long the connection is timing out. I haven't had that problem but obviously it's going to be extremely dependent on the I/O speed to the device.

Are you running gogrepo from the external disk you are saving too ?
avatar
Kalanyr: Are you running gogrepo from the external disk you are saving too ?
Bingo, I am actually doing that because then I have the script/login cookies/manifest file always with the collection, no matter to which PC I connect it to (in order to update/download the collection), and also the file paths for gogrepo remain always the same. E.g. my "windowsgog.bat" file, that I run if I update the collection on a Windows machine, on that same USB HDD looks like this:

python gogrepoc.py update -lang en -os windows
python gogrepoc.py clean \gog\
python gogrepoc.py download \gog\
python gogrepoc.py verify \gog\

So I don't have to put the drive letter in the path (e.g. E:\gog\), in case the drive letter changes on different Windows PCs where I connect the USB HDD.

I guess I can try to run the script from the root (Linux) partition instead and see if it makes any difference... Not sure why that would affect it (as it works just fine in e.g. Windows, running the script from the USB HDD itself), but I guess it never hurts to try.
avatar
Kalanyr: I never tested NTFS on Linux (as non-read only mode still came with warnings when I was writing that part), I was assuming exFAT for a shared drive.

If you want to a shared drive between Linux/Mac/Windows I believe the best bet is exFAT with 1024K allocation size (preferably formatted on Windows).

However I don't think that's the issue here.
avatar
timppu: E,g, the gogrepoc clean seemed to work just fine on the NTFS drive, it moved obsolete files into !orphaned directory normally.

Also I have a recollection that I would have run gogrepo a long time ago on Linux, similarly using a NTFS formatted USB HDD... so I figure maybe that was back when it didn't yet have the preallocation? Or then I remember wrong...

So my main question is, is it possible to disable the whole preallocation (like Gede suggested by commenting out certain lines), or does it take more than that?

I guess then my only option is to reformat the USB HDD into ext4 or xfs and find a way to read those filesystems in Windows (should be possible I guess, with some extra driver).
Yes, you can comment out the linux pre-allocation lines and it won't hurt anything. I'm intending to include the option to turn pre-allocation off on Linux if I can't work out some way to identify the file system of the device it's writing to and only preallocate on exFAT/NTFS devices.
avatar
Kalanyr: Are you running gogrepo from the external disk you are saving too ?
avatar
timppu: Bingo, I am actually doing that because then I have the script/login cookies/manifest file always with the collection, no matter to which PC I connect it to (in order to update/download the collection), and also the file paths for gogrepo remain always the same. E.g. my "windowsgog.bat" file, that I run if I update the collection on a Windows machine, on that same USB HDD looks like this:

python gogrepoc.py update -lang en -os windows
python gogrepoc.py clean \gog\
python gogrepoc.py download \gog\
python gogrepoc.py verify \gog\

So I don't have to put the drive letter in the path (e.g. E:\gog\), in case the drive letter changes on different Windows PCs where I connect the USB HDD.

I guess I can try to run the script from the root (Linux) partition instead and see if it makes any difference... Not sure why that would affect it (as it works just fine in e.g. Windows, running the script from the USB HDD itself), but I guess it never hurts to try.
The error you're getting is an SSL error and it's not a write error (its a read error).

The error actually appears to be completely internal to SSL, so why the file system is making any different at all is puzzling me.

And the error report doesn't indicate it comes from a specific gogrepoc line, so it's more about identifying what's causing the issue rather than even being a temporary fix.
Post edited February 22, 2020 by Kalanyr
I have a synology nas with ext4 and run gogrepoc on it without issues. One thing though, I did a verify on my library and it marked 5 md5 hashes wrong. But I had to remove those files manually to redownload, it could be automated imho. Also the script probably should verify the files before moving them from the downloading folder to the final folders.
Also slowly I start to wonder what I will do after I run out of disc space. At the moment my windowd installers take up about 4 TB :/.