Thanks, then I'll try that.

Running the script from the main root partition (and pointing the download to the USB NTFS partition) made no difference, but I think your theory is probably right. Apparently the preallocation does not actually fail, it just takes a very long time in this case (ntfs-3g), like five minutes for a 4GB file, and during that time the login or connection times out or something.

I don't think preallocation ever takes that long on Windows, so I guess this is somehow related to the interaction with the ntfs-3g.

There's actually -clean and -delete flags for verify , each does what you would expect with files that fail the verification.

I thought about doing that with regard to verification but verifying large files is very slow , so I opted not too.

I've had that issue several times already, so far I've "fixed" it by just moving my GOG installer collection to an even bigger (USB) HDD. I first started with a 500GB USB HDD (2.5") for my GOG installers, then moved to a 2TB USB HDD (2.5"), and now my latest is a 5TB USB HDD (still (2.5")). I think my GOG installer collection is currently around 4TB in size (Windows/English installers only + extras).

I earlier requested for a feature in gogrepo that you could divide your collection into two or more parts, as in "download games starting with # to M to E:/path1/, and games starting with N to Z to G:/path2/" etc. Then you could keep your collection up to date on several hard drives.

When I originally ran out of space on my 2TB USB HDD, I started doing that same manually, ie. after gogrepo update I divided the manifest file into two (first manifest file having the first half of my GOG collection, and the second manifest file the rest), and then ran gogrepo clean/download/verify twice towards two different 2TB HDDs. It worked, but I always needed my to do that manual manifest file editing after update.

If you don't have a need to keep your installers on portable USB HDD(s) like I like to do, then I guess you could look into some RAID setup and/or using Linux LVM or similar, where you can just add new hard drives to the pool whenever you run out of space, and to the system it just looks like one filesystem/partition even though it is across several hard drives. I am unsure if Windows has something similar as LVM, I presume there is something where you can join several hard drives into what looks like one whole big-ass partition?

EDIT: As I suspected, it is possible on Windows too:

https://www.windowscentral.com/how-create-one-large-volume-using-multiple-hard-drives-windows-10

I've just never tried it myself.

EDIT2: This Windows 10 feature sounds more like LVM, ie. virtual storage pools across several hard drives:

https://www.windowscentral.com/how-use-storage-spaces-windows-10

Not sure if any of this is relevant to your NAS setup though...

That stuff goes over my head, but what I see as abnormal is that preallocating for a 4GB file takes close to five minutes. I don't think this is an USB speed or filesystem issue because on Windows, using that same USB NTFS HDD, the preallocation doesn't take anywhere near that long.

Even if there wasn't that SSL error after preallocation, I'd still want to get rid of that 5 minute preallocation waiting time per (4GB) file... :)

Ps. I forgot to mention that when I restarted the gogrepoc download right after the "failed" prellocation, the download did seem to start normally (without an SSL error) to the NTFS partition. It didn't have to wait for the preallocation at that time because apparently the files it was going to download were already preallocated under the !download directory from the earlier attempt. Or?

Yes, that'd be correct.

I guess the issue is the linux NTFS support being much slower than native windows then.

There's still the problem of the connection timeing out and not being caught and reconnected though. I'll have to look into that.

Yeah, at least for this preallocation. I don't recall noticing any speed issues when e.g. copying lots of data and big files to and from my Windows NTFS partitions and Linux ext4 partitions (ntfs-3g installed), but clearly there is some performance issue with preallocation with ntfs-3g.

Not sure if we are using the same version of gogrepo (I think mine should be the latest(?) gogrepoc.py (development?) version from Kalanyr, but then I haven't checked lately if there has been new versions), but maybe that points to wrong lines in my case.

Line 1619 on my gogrepoc.py looks like:

downloadslangs = [game_item for game_item in item.sharedDownloads if game_item.lang in valid_langs]

and it is under the section "# hold list of valid languages languages as known by gogapi json stuff" (that is line 1606 in my case).

So what does your line(s) 1619 and 1706 look like, so that I am looking at the right part of the code?

EDIT: I presume it should be somewhere under the "# downloader worker thread main loop", there are lots of lines there apparently related to preallocation so still trying to figure which parts can safely be commented out without affecting rest of the script.

I use two hdd's in raid1, back up windows + extras only. For now I'm ok, but I it's scary how fast they fill up. And I don't think I'll expand to raid 6 or similar, I don't think I want to find out the rebuild time for 5*9TiB

Ok leaving the windows stuff alone. I try to comment out these eight lines in three different places (the whole else path), I presume it should take care of it at least.

##else:
##if sys.version_info[0] >= 4 or (sys.version_info[0] == 3 and sys.version_info[1] >= 3):
##info("attempting preallocating '%d' bytes for '%s' using posix_fallocate " % (sz,downloading_path))
##with open(downloading_path, "wb") as f:
##try:
##os.posix_fallocate(f.fileno(),0,sz)
##except Exception:
##warn("posix preallocation failed")

Yep, with those 24 lines commented out, now it runs just fine even on my Raspberry Pi 4 (Raspbian).

Downloading at around 64 Mbits/sec to my external USB NTFS hard drive now, only about 563 GB left to download of new and changed files...

Looks like I will be dropping Python 2 support fairly soon. One of the required modules for the module I was using for accessing GitHub which I was using to work on implementing auto-updating has dropped support for Python 2 but in a very bad way due to the awkward situation that arose from the author having lost his PyPi credentials at one point, I doubt it'll get repaired either since Python 2 is EOL now.

Gotta say I didn't see that coming, though I probably should have.

I thought about using gogrepo as normal linux package (e.g when script itself is installed system-wide. You cant write to its installation folder as normal user - instead, script's data (manifest, cookie) is saved into $HOME/.local/config/gogrepo, and games get downloaded to $HOME/.local/share/gogrepo).

Hmmm. There are some weird usability issues with allowing arbitrary redirection due to how stateful the GOG library/Tokens are. I'll have to think about this. Maybe a folder for each GOG account, which contains the manifest and Token for that account , and that can be arbitrarily redirected.

I would absolutely love if gogrepo was altered to fit normal Linux package installation. I'd probably take the time to formally package it for Fedora and CentOS.