It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3861 posts)(3861 posts)
(3861 posts)
Pages:
1
50
98
99
100
101
102
150
200
250
258
This is my favourite topic
Posted May 10, 2018
avatar
Kalanyr: I wonder if GOG crashed during your manifest update. That shouldn't cause that kind of error but I'm at a bit of a loss otherwise. It's kind of hard to test for the situation "GOG decides to have a total meltdown in the middle of an update / download command"
True. I was just making sure that it wasn't a format change on GOG's side. If not, I'll wait a few days and do it again on Friday. I was just excited to test out my new internet connection since I upgraded it. I don't believe it's my net though, that's screamin' fast now.
Seems more likely that one of more of their servers are still off-line.
Post edited May 10, 2018 by paladin181
Posted May 10, 2018
avatar
Kalanyr: I wonder if GOG crashed during your manifest update. That shouldn't cause that kind of error but I'm at a bit of a loss otherwise. It's kind of hard to test for the situation "GOG decides to have a total meltdown in the middle of an update / download command"
avatar
paladin181: True. I was just making sure that it wasn't a format change on GOG's side. If not, I'll wait a few days and do it again on Friday. I was just excited to test out my new internet connection since I upgraded it. I don't believe it's my net though, that's screamin' fast now.
Seems more likely that one of more of their servers are still off-line.
Well looking at what you've posted more closely I can see the probable issue, you've got &amp ; (ignore the space I just need to break the encoding) in that download link in place of & which is clearly an encoding error (&amp ; is the HTML encoding not something that should appear in a URL under normal circumstances). The weird bit is that gogrepo doesn't use that format directly at all (for most files and agarest isn't one of the rare exceptions), and it's a gog side redirect , I wonder if GOG is sending the wrong information or if I'm logging that error wrong. Are you using Python 2 or 3 ? I'll run in python 3 for a while and see if it's just a logging error which is possible, though it would make your issue more mysterious.

On the bright side that means it probably isn't actually a manifest problem since that URL is never stored in the manifest.
Post edited May 10, 2018 by Kalanyr
Posted May 10, 2018
avatar
Kalanyr: Well looking at what you've posted more closely I can see the probable issue, you've got &amp ; (ignore the space I just need to break the encoding) in that download link in place of & which is clearly an encoding error (&amp ; is the HTML encoding not something that should appear in a URL under normal circumstances). The weird bit is that gogrepo doesn't use that format directly at all (for most files and agarest isn't one of the rare exceptions), and it's a gog side redirect , I wonder if GOG is sending the wrong information or if I'm logging that error wrong. Are you using Python 2 or 3 ? I'll run in python 3 for a while and see if it's just a logging error which is possible, though it would make your issue more mysterious.

On the bright side that means it probably isn't actually a manifest problem since that URL is never stored in the manifest.
I'm using Python 3, and This is the first time the manifest did this. My last full update with the same version only listed 1 "400" in the log file; game (400/593).
Posted May 15, 2018
I just recently had to switch to an external 4TB usb drive, due to lack of internal space, and didn''t move properly and ended up having to redownload everything i have (due to the auto delete in my script). Now i'm getting these errors:
04:25:39 | attempting preallocating '374828614' bytes for './!downloading/broforce/gog_broforce_2.3.0.4.sh' using posix_fallocate
04:27:00 | 35.6MB 0.0MB/s 1x bloodrayne/bloodrayne_soundtrack.zip
04:27:00 | 867.0MB 0.0MB/s 1x bloodrayne/setup_bloodrayne_gog-1_(17893).exe
04:27:00 | 2091.1MB 0.0MB/s 1x bloodrayne_2/setup_bloodrayne_2_1.0_(19403)-1.bin
04:27:00 | 666.67GB remaining
04:27:00 |
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 280, in recv_into
return self.connection.recv_into(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1715, in recv_into
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1538, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (104, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 302, in _error_catcher
yield
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 384, in read
data = self._fp.read(amt)
File "/usr/lib/python3.6/http/client.py", line 449, in read
n = self.readinto(b)
File "/usr/lib/python3.6/http/client.py", line 493, in readinto
n = self.fp.readinto(b)
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 285, in recv_into
raise SocketError(str(e))
OSError: (104, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/requests/models.py", line 745, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 436, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 401, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 320, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: OSError("(104, \'ECONNRESET\')",)', OSError("(104, 'ECONNRESET')",))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./gogrepo.py", line 1736, in worker
dlsz = ioloop(tid, path, response, out)
File "./gogrepo.py", line 1559, in ioloop
for chunk in response.iter_content(chunk_size=4*1024):
File "/usr/lib/python3.6/site-packages/requests/models.py", line 748, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: OSError("(104, \'ECONNRESET\')",)', OSError("(104, 'ECONNRESET')",))
04:27:00 |
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 280, in recv_into
return self.connection.recv_into(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1715, in recv_into
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1538, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (104, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 302, in _error_catcher
yield
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 384, in read
data = self._fp.read(amt)
File "/usr/lib/python3.6/http/client.py", line 449, in read
n = self.readinto(b)
File "/usr/lib/python3.6/http/client.py", line 493, in readinto
n = self.fp.readinto(b)
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 285, in recv_into
raise SocketError(str(e))
OSError: (104, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/requests/models.py", line 745, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 436, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 401, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 320, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: OSError("(104, \'ECONNRESET\')",)', OSError("(104, 'ECONNRESET')",))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./gogrepo.py", line 1736, in worker
dlsz = ioloop(tid, path, response, out)
File "./gogrepo.py", line 1559, in ioloop
for chunk in response.iter_content(chunk_size=4*1024):
File "/usr/lib/python3.6/site-packages/requests/models.py", line 748, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: OSError("(104, \'ECONNRESET\')",)', OSError("(104, 'ECONNRESET')",))
04:27:01 | attempting preallocating '4294196222' bytes for './!downloading/brutal_legend/setup_brutal_legend_1.0_(18996)-1.bin' using posix_fallocate
Is this some sort of server outage, or am I temp banned from downloading from gog?
Posted May 15, 2018
avatar
kohlrak: I just recently had to switch to an external 4TB usb drive, due to lack of internal space, and didn''t move properly and ended up having to redownload everything i have (due to the auto delete in my script). Now i'm getting these errors:

04:25:39 | attempting preallocating '374828614' bytes for './!downloading/broforce/gog_broforce_2.3.0.4.sh' using posix_fallocate
04:27:00 | 35.6MB 0.0MB/s 1x bloodrayne/bloodrayne_soundtrack.zip
04:27:00 | 867.0MB 0.0MB/s 1x bloodrayne/setup_bloodrayne_gog-1_(17893).exe
04:27:00 | 2091.1MB 0.0MB/s 1x bloodrayne_2/setup_bloodrayne_2_1.0_(19403)-1.bin
04:27:00 | 666.67GB remaining
04:27:00 |
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 280, in recv_into
return self.connection.recv_into(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1715, in recv_into
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1538, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (104, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 302, in _error_catcher
yield
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 384, in read
data = self._fp.read(amt)
File "/usr/lib/python3.6/http/client.py", line 449, in read
n = self.readinto(b)
File "/usr/lib/python3.6/http/client.py", line 493, in readinto
n = self.fp.readinto(b)
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 285, in recv_into
raise SocketError(str(e))
OSError: (104, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/requests/models.py", line 745, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 436, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 401, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 320, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: OSError("(104, \'ECONNRESET\')",)', OSError("(104, 'ECONNRESET')",))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./gogrepo.py", line 1736, in worker
dlsz = ioloop(tid, path, response, out)
File "./gogrepo.py", line 1559, in ioloop
for chunk in response.iter_content(chunk_size=4*1024):
File "/usr/lib/python3.6/site-packages/requests/models.py", line 748, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: OSError("(104, \'ECONNRESET\')",)', OSError("(104, 'ECONNRESET')",))
04:27:00 |
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 280, in recv_into
return self.connection.recv_into(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1715, in recv_into
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1538, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (104, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 302, in _error_catcher
yield
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 384, in read
data = self._fp.read(amt)
File "/usr/lib/python3.6/http/client.py", line 449, in read
n = self.readinto(b)
File "/usr/lib/python3.6/http/client.py", line 493, in readinto
n = self.fp.readinto(b)
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 285, in recv_into
raise SocketError(str(e))
OSError: (104, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/requests/models.py", line 745, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 436, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 401, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 320, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: OSError("(104, \'ECONNRESET\')",)', OSError("(104, 'ECONNRESET')",))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./gogrepo.py", line 1736, in worker
dlsz = ioloop(tid, path, response, out)
File "./gogrepo.py", line 1559, in ioloop
for chunk in response.iter_content(chunk_size=4*1024):
File "/usr/lib/python3.6/site-packages/requests/models.py", line 748, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: OSError("(104, \'ECONNRESET\')",)', OSError("(104, 'ECONNRESET')",))
04:27:01 | attempting preallocating '4294196222' bytes for './!downloading/brutal_legend/setup_brutal_legend_1.0_(18996)-1.bin' using posix_fallocate
avatar
kohlrak: Is this some sort of server outage, or am I temp banned from downloading from gog?
I can't tell from that for certain , it's a generic "connection reset" error, it's probably not on GOG side though because they tend to block with (incorrect) Server response codes.
Posted May 15, 2018
By the way, turns out my problem was most likely on the gog end. My most recent download had few errors rather than being clogged with them the other day.
Posted May 15, 2018
avatar
kohlrak: I just recently had to switch to an external 4TB usb drive, due to lack of internal space, and didn''t move properly and ended up having to redownload everything i have (due to the auto delete in my script). Now i'm getting these errors:

Is this some sort of server outage, or am I temp banned from downloading from gog?
avatar
Kalanyr: I can't tell from that for certain , it's a generic "connection reset" error, it's probably not on GOG side though because they tend to block with (incorrect) Server response codes.
I think it might be the preallocate, actually. I decided to try to google see if the problem was mentioned before you took over. Someone before was complaining about NTFS drives and preallocate being slow, which is another problem i'm dealing with. I'm thinking that because it's over USB, it's taking so long, because it's not actually failing. And i think you're still trying to use the same connection that you got the download sizes with, which, since the preallocation is slow, the connections time out long before it's trying to download. As far as gog is concerned, it's like i'm trying to use slowloris DDoS attack on them, which is a DDoS attack where you try to flood with open connections that you never use to attempt to reach thread limit.

I remember reading that you felt it pointless to make it optional, because it would just continue if it failed. Well, it's not actually failing, just being slow, which causes it to fail. So off hand, what line could i replace or comment out to skip the pre-allocation?

EDIT: The conclusion came from doing "killall gogrepo.py" and now it's downloading everything that it was able to pre-allocate in !download.
Post edited May 15, 2018 by kohlrak
Posted May 15, 2018
I'm actually someone that would like for preallocation to be optional. Preallocation doesn't work for me as I download directly to a network connected NAS and so my logs show all of the preallocation errors making it harder to review them for other issues.

Also, surely it would be quicker in my case for preallocation to be off as it's one less thing gogrepo is trying to do.
Post edited May 15, 2018 by ikrananka
Posted May 15, 2018
avatar
ikrananka: I'm actually someone that would like for preallocation to be optional. Preallocation doesn't work for me as I download directly to a network connected NAS and so my logs show all of the preallocation errors making it harder to review them for other issues.

Also, surely it would be quicker in my case for preallocation to be off as it's one less thing gogrepo is trying to do.
where do you see ynur errors?
Posted May 15, 2018
avatar
ikrananka: I'm actually someone that would like for preallocation to be optional. Preallocation doesn't work for me as I download directly to a network connected NAS and so my logs show all of the preallocation errors making it harder to review them for other issues.

Also, surely it would be quicker in my case for preallocation to be off as it's one less thing gogrepo is trying to do.
avatar
kohlrak: where do you see ynur errors?
During download, I see the errors and they are also in the gogrepo log files. Here's an example:

08:48:04 | preallocating '62726499' bytes for '\\NAS\GOG Games\!downloading\tyranny_game\tyranny_ost_mp3.zip'
08:48:04 | could not get filehandle
08:48:04 |
Traceback (most recent call last):
File "gogrepo.py", line 1631, in worker
raise OSError()
OSError
08:48:04 | preallocation failed

This error happens for every single file just before it is downloaded.
Post edited May 15, 2018 by ikrananka
Posted May 15, 2018
avatar
kohlrak: where do you see ynur errors?
avatar
ikrananka: During download, I see the errors and they are also in the gogrepo log files. Here's an example:

08:48:04 | preallocating '62726499' bytes for '\\NAS\GOG Games\!downloading\tyranny_game\tyranny_ost_mp3.zip'
08:48:04 | could not get filehandle
08:48:04 |
Traceback (most recent call last):
File "gogrepo.py", line 1631, in worker
raise OSError()
OSError
08:48:04 | preallocation failed

This error happens for every single file just before it is downloaded.
when i get home, i'll take a look at line 1631. should be able to suppress that from there and make it optional.
Posted May 16, 2018
avatar
kohlrak: when i get home, i'll take a look at line 1631. should be able to suppress that from there and make it optional.
Line 1631 simply raises the error. I'd like to cut out / make optional the entire preallocation routine. I just don't know the extent of the lines that need to be deleted.
Posted May 16, 2018
Pre-allocation is only done for files immediately before they download (it doesn't pre-allocate everything first).
And your problem isn't soley due to a network timeout while waiting for pre-allocation, that would recover like any other timeout does, though it may be a contributing factor in a more complex problem.

I very strongly recommend against removing pre-allocation on NTFS / FAT variant drives as the normal behaviour will lead to extreme file fragmention over time which is why it was introduced (if you must you should also cut simultaneous downloads to 1 and try to avoid other write operations on the drive at the same time, it's probably even wise to redirect the logging to another disk).

Removing it on Linux filesystems should generally be fine.

I'll see about adding some options to skip pre-allocations (with appropriate warnings for NTFS / FAT ) and doing it otherways (like doing all pre-allocation first and then connecting to start the download sequence).
Post edited May 16, 2018 by Kalanyr
Posted May 16, 2018
avatar
Kalanyr: Pre-allocation is only done for files immediately before they download (it doesn't pre-allocate everything first).
And your problem isn't soley due to a network timeout while waiting for pre-allocation, that would recover like any other timeout does, though it may be a contributing factor in a more complex problem.
It's not retrying connections, from what i can tell. I think you're underestimating the length of the preallocation method. We're seeing things like 10 minutes or longer in my logs.

I very strongly recommend against removing pre-allocation on NTFS / FAT variant drives as the normal behaviour will lead to extreme file fragmention over time which is why it was introduced (if you must you should also cut simultaneous downloads to 1 and try to avoid other write operations on the drive at the same time, it's probably even wise to redirect the logging to another disk).

Removing it on Linux filesystems should generally be fine.

I'll see about adding some options to skip pre-allocations (with appropriate warnings for NTFS / FAT ) and doing it otherways (like doing all pre-allocation first and then connecting to start the download sequence).
Fragmentation can be fixed, though, and isn't a big deal since all it is is installers. The bigger picture is that, odds are, it's going to be NTFS drives that have problems with this to begin with. However, it's not like we aren't going to be running this script more than, maybe, once per week. At that rate, if linux has a defrag command in the NTFS tools, that might actually be appropriate to add to our update scripts (and on windows systems, anymore, defrag is scheduled weekly as a default setting).
avatar
kohlrak: when i get home, i'll take a look at line 1631. should be able to suppress that from there and make it optional.
avatar
ikrananka: Line 1631 simply raises the error. I'd like to cut out / make optional the entire preallocation routine. I just don't know the extent of the lines that need to be deleted.
It is there, actually. I'm no expert on python, but by the looks of things, you're using windows (corroborating my theory of NTFS drives being the ones most likely to have preallocation issues), and deleting 1630 through 1656 willl work as a temporary solution, so long as it is capable of working without preallocation. I'm about to try it now.

EDIT: there's alot more to delete, and due to the complexity, i'll just say that the stuff is near that line, and if you don't feel confident doing it yourself, you probably shouldn't be doing it.

EDIT AGAIN: General rule is to defragment anyway. The preallocate will still fragment after enough updates take place. You'll want to regularly defrag after running the script.
Post edited May 16, 2018 by kohlrak
Posted May 16, 2018
avatar
Kalanyr: Pre-allocation is only done for files immediately before they download (it doesn't pre-allocate everything first).
And your problem isn't soley due to a network timeout while waiting for pre-allocation, that would recover like any other timeout does, though it may be a contributing factor in a more complex problem.
avatar
kohlrak: It's not retrying connections, from what i can tell. I think you're underestimating the length of the preallocation method. We're seeing things like 10 minutes or longer in my logs.

I very strongly recommend against removing pre-allocation on NTFS / FAT variant drives as the normal behaviour will lead to extreme file fragmention over time which is why it was introduced (if you must you should also cut simultaneous downloads to 1 and try to avoid other write operations on the drive at the same time, it's probably even wise to redirect the logging to another disk).

Removing it on Linux filesystems should generally be fine.

I'll see about adding some options to skip pre-allocations (with appropriate warnings for NTFS / FAT ) and doing it otherways (like doing all pre-allocation first and then connecting to start the download sequence).
avatar
kohlrak: Fragmentation can be fixed, though, and isn't a big deal since all it is is installers. The bigger picture is that, odds are, it's going to be NTFS drives that have problems with this to begin with. However, it's not like we aren't going to be running this script more than, maybe, once per week. At that rate, if linux has a defrag command in the NTFS tools, that might actually be appropriate to add to our update scripts (and on windows systems, anymore, defrag is scheduled weekly as a default setting).
avatar
ikrananka: Line 1631 simply raises the error. I'd like to cut out / make optional the entire preallocation routine. I just don't know the extent of the lines that need to be deleted.
avatar
kohlrak: It is there, actually. I'm no expert on python, but by the looks of things, you're using windows (corroborating my theory of NTFS drives being the ones most likely to have preallocation issues), and deleting 1630 through 1656 willl work as a temporary solution, so long as it is capable of working without preallocation. I'm about to try it now.
Large collections on NTFS drives are probably USB HDDs, these are not generally defragmented on Windows. I'll provide the option for those who wish to disable it, I just know from experience it's a really bad idea in the case of non-network NTFS drives, my last GOG archive drive before the rewrite had massive fragmention (and took over 2 days to defragment).

Your solution will work, though you should also delete the analogous section following the next else (as it does pre-allocation for new files which is the most common case)

Defragmention over time due to updates will take a time approaching infinity to match 4 threads writing small chunks of data to first available space simultaneously. Especially since most of GOGs disk use comes from files that are exactly 4 Gigabytes in size.
Post edited May 16, 2018 by Kalanyr
Pages:
1
50
98
99
100
101
102
150
200
250
258
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3861 posts)(3861 posts)
(3861 posts)