timppu: I recall there was some issue with the gogrepo file preallocation in Linux, what was the issue again?
So far I've been mostly running gogrepoc in Windows, but I tried to run it on Linux now (both on my Mint PC and Raspberry Pi). The important point is that the USB HDD to which the installers are downloaded is NTFS, as the games will be installed in Windows anyway. Naturally I have installed ntfs-3g in Linux so that it can read and write to NTFS partitions just fine.
python3 gogrepoc,py update & clean seem to work just fine, but when gogrepoc tries to download, the "preallocation" takes several minutes, and after that I start getting odd errors. The screen capture is at the end of this messge.
Also, I just tested by redirecting the downloads to an ext4 Linux-partition instead... and sure enough, the downloads start just fine and the preallocations also take mere seconds, instead of several minutes. (I recalled using gogrepo earlier in Linux just fine, so I gather either I was directing the downloads to a Linux partition, or the gogrepo version was so old that it didn't yet have the preallocation feature?)
If it is so that the preallocation feature is incompatible with NTFS in Linux (ntfs-3g), is it possible to disable the file preallocation in the script easily, or is it so integrated to everything that it is hard to get rid of? I was never a big fan of that preallocation anyway, seems kinda pointless if you ask me (fragmentation schragmentation, who cares, these are big files).
I guess the other option is that I reformat the USB HDD where I keep my GOG installers to ext4 or xfs and then use some driver in Windows so that Windows can read ext4/xfs partitions, but for simplicity's sake, I'd rather keep it NTFS.
This is what it looks like when running python3 gogrepoc.py download to a NTFS partition (notice the timestamps how long each preallocation takes):
21:18:36 | ------------------------------------------------------------
21:18:36 | attempting preallocating '109576656' bytes for '/media/user/GOG/GOG/!downloading/80_days/setup_80_days_1.17.6_(35219).exe' using posix_fallocate
21:18:42 | attempting preallocating '4293628414' bytes for '/media/user/GOG/GOG/!downloading/a_plague_tale_innocence/setup_a_plague_tale_innocence_1.07_(64bit)_(34188)-1 .bin' using posix_fallocate
21:23:30 | attempting preallocating '4294967294' bytes for '/media/user/GOG/GOG/!downloading/a_plague_tale_innocence/setup_a_plague_tale_innocence_1.07_(64bit)_(34188)-2 .bin' using posix_fallocate
21:28:21 |
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 302, in _error_catcher
yield
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 384, in read
data = self._fp.read(amt)
File "/usr/lib/python3.6/http/client.py", line 459, in read
n = self.readinto(b)
File "/usr/lib/python3.6/http/client.py", line 503, in readinto
n = self.fp.readinto(b)
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.6/ssl.py", line 1012, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.6/ssl.py", line 874, in read
return self._sslobj.read(len, buffer)
File "/usr/lib/python3.6/ssl.py", line 631, in read
v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer
It's normal for preallocation to take much longer on exFAT/NTFS systems as they really do the pre-allocation (so they are writing 4 GB of data here) , ext3/4 don't actually have the capacity to do true pre-allocation (and it's not really necessary because they are much less subject to fragmentation). Best cross system compatibility in my experience is exFAT with 1024K allocation size formatted on Windows.
Those errors don't look related to the file system, they are SSL connection errors. They also aren't related explicitly to a gogrepoc line, so it's something happening to an existing connection that's handling things on its own.
I wonder if it's taking so long the connection is timing out. I haven't had that problem but obviously it's going to be extremely dependent on the I/O speed to the device.