Following up on this, turns out the Python request library follows redirects by default for GET/POST (and I assume PUT ) but not for HEAD for some reason, but there's an allow_redirects that parameter to override the default behaviour.

Well the fix would not have retroactively created blue dots for everything that was ever missed so when's the last time you did a full scan before January? What games exactly did you discover? Also are you sure the changes were with the games and not just extras? Extras changes still do not generate blue dots. Never did.

That would explain the different results that you were seeing.

Beginning of December.

I did notice a noticeable dropoff though but I'd put it down to fewer updated being done in December.

Hmmmm. Extras could well be it. I'll have to see what happens on my next full update which should be this week sometime

I doubt this is going to effect anyone at this point but I'm going to do some cleaning up and move to (mostly) only supporting non-EOL Python versions.

For now it just means that I'll only support Python 2.7 , nothing older than that and I'll only support non-EOL Python 3 variants (except I'll maintain 3.8 specific backwards compatibility for now since it's the last Windows 7 version), (for now this means I support 3.8+ since 3.9 isn't EOL yet )

I will at some point in the future completely drop 2.7 but I really wanna get the new manifest format / folder setup done before I do that.

Likewise I'll drop 3.8 eventually but that's less of a hassle than 2.7 back.

Is anyone using Python 3.8-3.10 ? Particularly if you're using Mac or Linux. I may require some testing on stuff unique to that version range with the next release.

Okay, got all these done on my local copy. Have a few more things to do ( fixing the encoding issue with some serial keys (WC2), orphaning updating image files instead of nuking them and seeing if I can test the validity of archive sets (like the ZIP files used for the Mac version of Bards Tale IV director's cut ) and then testing this stuff with a full update and I should be ready to roll this out for testing.

I did implement the FS detection but it's going to require an additional module psutil , I also learnt that MacOS does not support fallocate , what does POSIX complaint even mean ? Is it a hallucination ? Which means preallocation should fail much more rarely on Linux now, it only tries preallocation on the fs types explicitly supported by the linux kernel + exFAT + vFat (which is likely to only tag FAT32 in practice ), NTFS should theoretically be allowed but I've blocked that given the reported strange handing behaviour on some network drives. Windows will also not attempt preallocation on Linux FS if that comes up for some reason (maybe you have a NAS mapped to a local drive handle or something ? or you're using a 3rd party driver )

ETA - I think I'll finally make the new/separate thread with this release too. I should probably update the documentation too.

Wait a second, are you saying that you're stopping preallocation on NTFS? Please no, just add an option to disable it for those few who use network drives!

And it's great that you're finally creating your own thread!

Only Linux + NTFS, still doing it on Windows.

Hmmmm, you make a good point though, I should reenable it for now , because it's possible the problem was trying to preallocate on NTFS being reported as NFS which no longer attempts preallocation anyway.

I am using Python 3.9 in Linux.
https://github.com/niess/python-appimage

Sent you a Friend Invite.

It happens also for me with using an USB hard drive, no NFS.

So the USB hard drive is formatted to NTFS, and if I run gogrepoc on Linux, trying to download to that USB hard drive, preallocation fails, every time, on every Linux PC with various different Linux distributions that I've tried. Happens with any NTFS formatted USB hard drive I've used.

I am pretty sure it would happen even if the NTFS formatted hard drive was internal, not an external USB or a NFS hard drive. Something in the way Linux handles the NTFS partitions is apparently incompatible with the way the script does the preallocation.

It would be simpler if the user could disable preallocation at will, either in the script itself (set some variable to 0 instead of the default 1; the same way the user can change e.g. the number of download threads within the script (I prefer 2 or 3 instead of the default 4)), or with an option in the command. Then those who run into the problem for any reason, or want to disable preallocation for whatever reason, can easily do so.

EDIT: I haven't tested the latest gogrepoc.py though, so in case there is some change now where it will not try preallocation if it detects NTFS on Linux, I don't know yet it if works right now.

The script just calls the os.posix_fallocate , which in turns calls the OSs Posix Complaint fallocate. It's actually much simpler than the Windows pre-allocation.

I must try this out, I have a couple of hypotheses about what might be happening ( 1) Windows and Linux have in the past written slightly different partition data when initially formatting a partition so it might be hitting data it's not expecting and hanging 2) the kernel NTFS support is actually busted for fallocate, in which case you need to make a bug report to the appropriate maintainers )

Okay, found the issue with the WC2 serial code encoding, it's encoded in slightly malformed (missing the final padding character since it's an odd length byte sequence) UTF-16 for some inscrutable reason. Now the question is how to handle that remotely generically.

Okay, handled for Python 3, does anyone use Python 2.7 still ? Would appreciate some help with testing if you do, otherwise I guess I'm going to have setup a Python 2.7 environment.

ETA - In extremely cursed news Python 2.7 actually handles this better than Python 3.