It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
avatar
timppu: Update worked, clean worked... but download fails to "preallocation failed" problems.

The target device (an external USB hard drive) has plenty of free space (750GB) so it shouldn't be about that.

Am I supposed to have the target path on the same partition where I run gogrepo? I run gogrepo on my local C: partition (which doesn't have much of free space, 11GB), but the target where to download the games is on an external USB hard drive (I:).

*snip*
avatar
timppu: etc. etc.
Sorry incoming question stream (I suspected this may happen as the preallocation commands are OS level rather than native python). and there's potentially some issues with the changes between strings in Python 2 /3 as well.

What python version are you on ? What OS and version are you on ? And what File System is the disk.

I suspect the difficulty is that you're Windows versoin is expecting a path prepending with \\?\ but that breaks recent Windows 10 versions with MAX_PATH_LIMIT off.

The path at least seems correct , so it's probably not an issue with the script location. Unfortunately this is obviously very difficult to test without an appropraite OS, I'll set up a virtual machine in the next week and see what happens.

In the mean time preallocation failing shouldn't be any worse than it was before preallocation was added.
Post edited December 04, 2017 by Kalanyr
avatar
Kalanyr: What python version are you on ? What OS and version are you on ? And what File System is the disk.
Windows 7 64 bit, Python 3.5.2 (32bit), NTFS on all partitions (including the external USB disks).

EDIT: I see the latest python version is now 3.6.3, could it be about that then?

EDIT2: It also seems you updated gogrepo right after I started using it, but not sure if it affects this problem.
Post edited December 03, 2017 by timppu
avatar
Kalanyr: What python version are you on ? What OS and version are you on ? And what File System is the disk.
avatar
timppu: Windows 7 64 bit, Python 3.5.2 (32bit), NTFS on all partitions (including the external USB disks).

EDIT: I see the latest python version is now 3.6.3, could it be about that then?

EDIT2: It also seems you updated gogrepo right after I started using it, but not sure if it affects this problem.
No, the pre-alloc stuff is either present in 2.7+ or 3.3+ so 3.5.2 should be fine.

No, none of my changes would have affected preallocation.

It's likely the different way Windwos 8 handles paths that violate MAX_PATH_LIMIT , the latest few windows 10 versions will handle it transparently. I'll see what I can do about massaging the string for maximum compatability.
So it is not possible to make preallocation optional? Does this affect all Windows 7 users?

I personally never cared for preallocation (on gogrepo). If I want to avoid fragmentation on my GOG game installers, I'd download using only one download thread, and maybe occasionally defragment the storage drive. And since only a small part of my existing GOG installers get any updates anyway, it is not like the storage drives are constantly getting very fragmented.

I've used the previous versions (other forks) of gogrepo this long without much of issue with fragmentation. I should be much more concerned about the fragmentation of my OS partition, than that of my GOG installer partition which I use rarely (only when I install new games) and which contains mostly just huge files.
Post edited December 03, 2017 by timppu
avatar
Kalanyr: In the mean time preallocation failing shouldn't be any worse than it was before preallocation was added.
I am unsure what this means, but gogrepo doesn't download anything for me. It just reports lots of "preallocation failed" errors and then exits.

Before preallocation was added, at least I could download the installers. :) I was using the vanilla version up until now, until the %-codes problem surfaced.
avatar
Kalanyr: In the mean time preallocation failing shouldn't be any worse than it was before preallocation was added.
avatar
timppu: I am unsure what this means, but gogrepo doesn't download anything for me. It just reports lots of "preallocation failed" errors and then exits.

Before preallocation was added, at least I could download the installers. :) I was using the vanilla version up until now, until the %-codes problem surfaced.
That is not normal or in line with my testing. The preallocation is done indepently of the download loop. So there must be another defect tcausing that on your setup.
Obsolete message, problem fixed.
Post edited December 04, 2017 by timppu
avatar
Kalanyr: That is not normal or in line with my testing. The preallocation is done indepently of the download loop. So there must be another defect tcausing that on your setup.
avatar
timppu: The log file didn't reveal anything, the last few lines are at the end of this message,

My first thought was that I was running out of space on that target partition, but it has plenty of space. I was also running gogrepo (the command prompt) as administrator, as I normally do. I also tried it on my other USB HDD (which has the other half of my GOG installer collection), same results. So it isn't about the specific USB HDD either, it happens on both.

I am unsure what to test next or where to look for the culprit. The vanilla gogrepo (without preallocation) worked fine the last time I used it on this same setup. EDIT: Maybe I'll do some testing downloading single games with gogrepo, e.g. to the same local partition where gogrepo is running, just to see that the problem happens everywhere.
*snip*
avatar
timppu:
I don't understand why you're not getting a message about why the download is failing. Can you try this and let me know if it fixes the preallocation error at least:

https://www.dropbox.com/s/a89ewreqo05pdef/gogrepo.py?dl=0

(Please do not try this at random, I'll integrate it if it does work, and if it doesn't , have fun tracking down where it put the file it was trying to open).
Post edited December 04, 2017 by Kalanyr
avatar
Kalanyr: I don't understand why you're not getting a message about why the download is failing. Can you try this and let me know if it fixes the preallocation error at least:

https://www.dropbox.com/s/a89ewreqo05pdef/gogrepo.py?dl=0

(Please do not try this at random, I'll integrate it if it does work, and if it doesn't , have fun tracking down where it put the file it was trying to open).
What do you mean not trying it at random? Not try it to my whole GOG collection, but only to download a single game, or...?
avatar
Kalanyr: I don't understand why you're not getting a message about why the download is failing. Can you try this and let me know if it fixes the preallocation error at least:

https://www.dropbox.com/s/a89ewreqo05pdef/gogrepo.py?dl=0

(Please do not try this at random, I'll integrate it if it does work, and if it doesn't , have fun tracking down where it put the file it was trying to open).
avatar
timppu: What do you mean not trying it at random? Not try it to my whole GOG collection, but only to download a single game, or...?
It was a warning to other people who think it might be a good idea to randomly try a variant targeted at your situation.
But yeah ideally delete your !downloading folder , and then just download 1 game that has at least 5 items to download (just move the existing files somewhere safe).
Post edited December 04, 2017 by Kalanyr
A bit more info about the failed run: at least it seems the script has created a !downloading directory to the target partition, the per-game subdirectories for the files themselves it is going to download, and "touched" the files so that now there are lots of 0 kB files under !downloading.

So at least the script seems to be able to write something to the target partition. Well, since clean worked fine, it was also able to move files around (to the !orphaned-directory).

EDIT: I will try the test version later...
Post edited December 04, 2017 by timppu
avatar
timppu: A bit more info about the failed run: at least it seems the script has created a !downloading directory to the target partition, the per-game subdirectories for the files themselves it is going to download, and "touched" the files so that now there are lots of 0 kB files under !downloading.

So at least the script seems to be able to write something to the target partition. Well, since clean worked fine, it was also able to move files around (to the !orphaned-directory).

EDIT: I will try the test version later...
Unfortunately without knowing why the download loop failed the 'touched' files aren't very useful, as the first thing the download loop does is create the files if they don't exist. It's also possible that it is the preallocation and it isn't getting a proper file handle back. Have you got any extra random files in the directory you were running gogrepo from ? I'd expect it to be a single file named g if you are.
Post edited December 04, 2017 by Kalanyr
Obsolete message, problem fixed.
Post edited December 04, 2017 by timppu
If anyone using Windows 8 / 8.1 having problems with the pre-allocations ?

avatar
timppu: Not sure if I should take this to PM in order not to clutter this thread with problem solving, but then if someone else has something to add (mostly I am wondering if I am the only Windows 7 user having this problem?).

Anyway, for the test setup, I made these arrangements so that there are less moving parts:

1. I downloaded your test version
2. I try to download a single game (Necrovision) to my local C: partition where I run gogrepo, not the external USB partition.
3. I use the original full manifest file that I got yesterday, not the parts I divided into two manually (just so that I didn't accidentally make some mistake while dividing the manifest).

The test version still fails.
*snip*
Try this version please : https://www.dropbox.com/s/a89ewreqo05pdef/gogrepo.py?dl=0 it's got additional debugging info. You'll need to delete your !downloading directory again.

(I'm sort of wondering if Windows 7 has CreateFileW , I do have a messier version using subprocess I was using before I decided to handle things programmatically I can fall back on if that's the case. . )
Post edited December 04, 2017 by Kalanyr
avatar
timppu: (mostly I am wondering if I am the only Windows 7 user having this problem?)
Hey, I have Win7 too.
I could make a test with a game in a temp folder, but please tell me the commands to use.