It's all the missing updates that didn't get flagged. I only got 14 GB of real updates out of it.

Is the GitHub repo in the OP still the current, or is some fork preferred?

The latest and greatest is Kalanyr's fork which can be found here.

I'm already a couple hundred gig into my couple TB download with the original. Hopefully the new version can import that. Looking now. Thanks!

If you simply replace the script in the location you pot the old one, it will work fine as long as you use the correct commands.

I haven't updated gogrepo in a while (because I require personal fixes to the code, which are annoying to do), so i found a "bug" i felt i shouldn't report without updating first, but having my own personal fix (deleting the DB files), i didn't report it. I finally decided to come back and skim to see what has changed since i last updated, and it seems there might not have even been an update (right now reporting k0.3a). Therefore, i'm going to report it.

If you buy a game, it goes in the DB no problem. If you buy DLC, goodies, or whatever for a game that's already in the DB, since it doesn't get marked updated, and since it's for a game already in the database, it ignores it, thus you can end up with alot of stuff missing from the updates.

Maybe the old manifest file should be removed too and a full update to be run (with the new script), or are the manifest files compatible?

I delete the old manifest file every time anyway, so...

I presume this happens only if you keep the old manifest file (when running update)?

To me that wouldn't really seem like a bug in gogrepo, it is merely a "feature" in the GOG store (ie. the games for which you buy new DLC don't get marked as "new" (or "updated", for that matter) in your account.

At least the workaround is rather simple in gogrepo, I believe: delete the old manifest file and run gogrepo update, because then you are not relying on those "new" or "updated" flags, but you are guaranteed to receive any changed or added files. Maybe there was also already a separate gogrepo option to do a full update, even if you have an existing manifest file...

this has always been the case, and that is what the full update is for. the updatesonly option can only check what has been updated by GOG, and since they do not flag the changes you mention, no luck there.

Yes, that's correct, GOG's tagging for updates is mediocre, they frequently don't mark things that were updated as updated (though they've gotten a bit better at this lately) and they don't mark things you've got new DLC for as updated.

Yeah, the default on the current normal branch is full update with updateonly / skipknown requiring specification.

On the dev branch I've changed things to get new games + updated games by default , with the full update as an option (it seems kinder on GOG's servers to make the normal and far more efficient behaviour the default) with some smart adjustments (does a full by default if know manifest exists and scans everything when you specify specify -ids, though these can be overridden).

(I do not recommend deleting the manifest. If the manifest is casuing problems I need to know so I can fix it. There should literally never be a reason to delete your manifest on my fork, other than wanting to use a different GOG account.. A full update should place the manifest in the same state as it would be in after deleting the manifest and letting it do a full update. )

I apologies for the very out of date documentaiton on my fork, I've got a few bits and pieces I want to finish on the Dev branch and roll back into master which will break some of the syntax, so I want that nailed down before I update the docs.

It actually didn't work as a drop-in replacement. I'll check out more over the weekend when I have time again. In the meanwhile, my gigabytes are downloading with the eddie one. It took forever to build the manifest (i have 1140 products), so I didn't want to start that over.

Any chance we can get the mods to edit the OP to point to a different one indicating the newer version?

How about mqstout's case where he has already created a manifest file using the original wooly's gogrepo, and going to switch to yours? Your fork handles wooly's manifest file fine, as long as one runs a full update?

I usually delete the old manifest file due to laziness. I am not even sure if I am using the normal or dev branch version (ie. what is the default behavior, does it do a full or "flagged games only" update by default) etc. At least I know what I am getting when starting it clean each time. :) Then again I alleviate that by not doing it often, usually once a month or even less.

Plus, since I have the special case that I have to divide the manifest file into two after the update (because my GOG game installers don't fit into one hard drive), I don't get much of the other benefits like skipping unaltered files in verification etc. (unless I'd keep dividing and joining the manifest file depending if I am going to run an update, or download/verification).

Kalanyr's version runs the update much faster than wooly's old version, even if starting from the scratch. Something like 1/5 of the time maybe, so if the update took you 10 hours with wooly's version, it might take "only" two hours with kalanyr's version.

Doesn't the old version have the problem with some of those special characters in filenames (which was later fixed in the kalanyr's version)? That might mean you end up downloading many of the files all over again with kalanyr's version because currently they will (possibly) be named incorrectly with your downloads. There was some discussion earlier in this thread like over a year ago or so when it occurred, I don't quite recall the details anymore... So in that sense I'd personally switch to kalanyr's version right away.

EDIT: It is this problem:

https://www.gog.com/forum/general/gogrepopy_python_script_for_regularly_backing_up_your_purchased_gog_collection_for_full_offline_e/post1337

And since almost all installer files and some extras files have such special characters like ( ) &, I propose you stop the download with Ctrl-C. check if you have that issue with your downloaded files' filenames, and move to using kalanyr's version.

Probably easier that kalanyr just starts a new thread and this thread eventually dies.

Thanks. Cancelled and starting over. Verizon's going to hate me for the wasted 1 TB, but I was seeing the broken characters.

This is a very lengthy thread, which is why I asked. Reading 30+ pages (and I have my page size set to max!) is burdensome. Thanks for the help, everyone.

Hopefully this also will be good at updating to find new things, though I read that, with DLC, you have to get a new manifest. Will downloading later result in keeping duplicates (the old versions)? It looks like the "cleanup redundant files" was left in the wishlist but not implemented.

Oops, 1 terabyte already, he he. I am unsure what is the most optimal way to proceed, but if I were you, I would probably do something like this:

1. Switch to using the kalanyr version of the script. Not sure if you can keep using the old manifest file and e.g. your existing login cookies, but since the manifest file now has buttloads of wrongly named files, I'd probably just remove it and start the update all over.

2. After running update, run the "gogrepo.py clean -dryrun" command on your existing download directory to see how much would be scrapped and downloaded all over again.

3. Then make the decision if you want to clean (without -dryrun) and delete all those wrongly named installers and redownload them, OR try to rename them somehow as that would probably save you lots of redownload time because I think the already downloaded files should otherwise be ok, just with wrong filenames, as far as I know.

4. Whichever you chose, after downloading everything run the "gogrepo verify" to make sure you now have on ok set.

Are you Windows or Linux or what? I guess renaming buttloads of files is possible in both (in Windows you'd probably want to use PowerShell, or I don't know if there is some 3rd party Windows utility to rename lots of files), but at least in Linux I'd just google for "linux rename many files recursively" or "linux change many filenames recursively", I'm sure you'll see lots of suggestions for a nice bash script or command to do that.

I don't recall what were all the special character you'd see in your GOG installers and extras, but I think the most common ones are ( and ). Some games have also & but IIRC it is mostly some extras files for a few games, not that many. Not sure if some files had also e.g. ' but I think the ( ) are the most important to fix.

So IIRC these are what should be changed in all filenames. I don't remember these by heart, I checked them from here:

https://www.w3schools.com/tags/ref_urlencode.asp

%26 change to &
%28 change to (
%29 change to )

You can of course run the command (e.g. mv or rename in Linux bash) three times so that it first changes the %26, then %28 and then %29. Just make sure you run the command/script recursively of course, as all the GOG installers are in separate subdirectories.

If the bash/powershell suggestions by google go over your head, please report here, there are enough experts here to help. I have some idea how I'd try to do it in bash (Linux), but even I'd have to confirm it with google. Many ways to do it of course, mv or rename with nice piping...

EDIT: I noticed the newest Linux Mint (Xfce) even comes with a graphical tool called "Bulk Rename" ("Rename Multiple Files"). Sweet! I am kinda surprised Linux would have such a tool by default and not Windows, as Linux users are generally more accustomed to doing things without a dedicated graphical tool, by using a couple of bash commands.


As for this thread, I presume the best way would be for kalanyr to just start a new thread, and put one message here mentioning this thread is obsolete and a link to the new discussion thread, and urging people to post messages only to the new thread?

Yes, the old manifests should work and be updated dynamically as necessary. If it doesn't that's a bug I need to fix.

Edit - Gogrepo has an import function which should match and rename files by md5 after you do a full manifest update (but before download), only where GOG provides md5s of course but that should be the really big stuff.

I just went clean.

I'm running that on my Linux box, but for renaming in Windows, this is one of the best utilities I've ever found: https://www.1-4a.com/rename/

I have noticed the kalanyr verion seems to download more slowly, but that's OK. It saves as you go return manifests, which is way, way better. And it was away faster getting the manifests!

Thanks, everyone!