Ah yes, thanks. I am at least somewhat familiar with SQL commands, if that helps with SQlite. :) I guess there will be still a way to edit it then, at least somehow. The other reason for me to edit the manifest file is to be able to divide it to two, so that I can use two different drives for the collection, half of the library for each drive.

One way of blacklisting files is to comment out the part of the script in the download section that checks file size then create a dummy file with the name of the file you don't want. That way it will pass on those files.

Does mean you have a lot of 0byte files and requires a bit of copy and pasting.

Specific lines are :
865 elif game_item.size != os.path.getsize(dest_file):
866 warn(' fail %s has incorrect size.' % game_item.name)

But then the script loses the ability to redownload partially downloaded files, right? (they would pass too?)

I guess I could live with that though, after all "verify" would still catch (and delete) such files anyway. That sounds quite an interesting workaround, I might try that.

EDIT: Ummm, right... wouldn't verify complain about those dummy files as well? So i guess "verify delete" option shouldn't be used then, but manually delete files which fail the verification (excluding the dummy files)?

Yeah there's some manual work involved there, but I kinda like your suggestion, it is indeed one way to kinda get rid of the files that you don't want (like the Russian translation of Planescape, or the Mac/Linux versions of Broken Sword Classic, in my case)...

The size check before download is usually needed if one tends to cancel the download process from time to time, because they need the bandwidth for something else or download the whole set in several sessions. I guess the manual workaround for that is to delete those partial files as soon as you abort the script (since you know at that point which files they are).

Come to think of it, I don't think I've used the delete option with verify really. I usually just run verify and then check myself from the list which files failed the check, and then delete them myself. I guess I am afraid there's some freaky hiccup with the script and it decides to delete most or even all of the files, even though they are fine. I don't like to easily give scripts the ability to delete files in their own discretion. I rather let them just suggest what to delete. :)

GOG has created something of a problem for the Downloader

The Walking Dead Michelonne
Hard West
Reigns

now have multiple files with the same name, but different file sizes / md5s. This leads to an infinite loop of downloads since its impossible to satisfy both.

I think the best way to handle this is to scan through the file names after a game entry is updated and automatically add an en enumeration to duplicate file names, which will also be used as the download name.

ETA - I've reported this to GOG but if I don't hear back from them this week, I might see if I can work out a temporary fix for this. I was willing to ignore it while it was just Reign but it's a never ending 11 GB download loop of suffering since it expanded this week. And the fact it expanded this week means it's likely going to continue expanding unless GOG fixes wtf is going on.

ETA2 - I've had a quick look at this, the good news is that this should be doable and I can see where. It also looks like it would be pretty easy. If anyone else is having issues with this I'll post my solution here next week if it becomes necessary.

When you talk about "Downloader", are you talking about the old GOG Downloader client (which is nowadays unsupported), or the gogrepo.py download tool that this thread is about?

I don't have any of those three games so I can't check if this would be a problem for gogrepo, but my understanding is that it is not a problem (for gogrepo).

My faint understanding (without checking the script itself) is that just before gogrepo starts the download process, it quickly cross-checks whether the files you have already downloaded have the same filename, and the exact same size, as what is available in the GOG servers (according to the manifest file). If either one is incorrect, it downloads the whole file all over again.

So, if GOG has a file that has the exact same name as before but different size, gogrepo simply redownloads the whole file. No problem. I actually presume this is quite often the case with Game Goodies, e.g. if some manual zip file has changed, its filename is not necessarily changed, only the size may change (albeit GOG seems to quite often change the filename too at the same time, but there is no version numbering for Game Goodies).

If it happened that the new file had the exact same filename and size as the old file you already have, I guess gogrepo would (incorrectly) just think you have the new file already, and skip downloading it. However, when you run "gogrepo verify", then that wrong file would be caught as it still has the wrong md5 checksum (regardless of its correct name and size), and after that it could be redownloaded.

(except for maybe game goodies, as they don't have md5 checksums...)

It seems the checksum of "The Walking Dead Michelonne" is not correct. I have problems with this game too. I use other scripts and tools based on the gog-manifes.dat. It seems the checksum in the gog-manifest.dat do not match with the checksum of the downloaded files (I hope there is no virus inside). In the last few days gog.com has replaced many installer files while maintaining the old file name (without increasing the version number). Very mysterious....

I guess GOG should be notified, hopefully they fix that. At least previously GOG fixed e.g. a problem where the language filter was wrong for Iron Storm (the English version was under the Polish language version, meaning that by default gogrepo wouldn't download anything for Iron Storm as it didn't find anything under the English language filter). GOG fixed it quite soon when I notified them.

For gogrepo users, I guess "gogrepo verify" will just complain that the file is incorrect, but beyond that it shouldn't cause problems I presume (ie. "gogrepo download" will not try to redownload it, as long as its filename and size is correct). So the user just needs to remember the installer is fine even though "gogrepo verify" says it is wrong (as it doesn't match the md5 checksum).

The checksums aren't wrong exactly , it's just that there's 2 copies of these files with different sizes and md5s. That means one or the other gets stomped at download. And it then fails the check for the other version when next you verify or download and gets stomped by the other version, which continues in an eternal cycle because a single file can't possibly satisfy two different sizes / md5s simultaneously.

I reported this to gog last week when it was just Reigns but it spread to Michellone and Hard West this week.

So at the same time two different files with the same name (for the given game)? Interesting.

I wonder how GOG differentiates them on their (CDN) servers, do they get downloaded from two different servers or directories? i presume GOG CDN servers can't keep two different files with exactly the same name in the same directory either?

They have different downlaod URL, so the same filename does not conflict in CDN.

I assume he meant the issue of them being stored on a server using the same filename, the actual file not the URL.
Assuming they use regular filenames of course.

The files wouldn't have the same path on the server (like being in different directories, they might also not even have the same filename when stored the filename to download them as could be stored as metadata and then passed along as part of the URL or download process), or might not even exist explicitly as files at all (they might be blob entries in a database, in which case they'd have different database entries).

A lot of what you see when interacting with a server isn't what's going on on the server, because so much of the web is dynamic these days.

I know, I was just explaining the reasoning behind the post.