With the current manifest file implementation, it is at least possible for the user to e.g. manually divide the manifest file into two (or more) separate manifest files, in order to download the games to several separate hard drives/partitions, in case not all of them fit into one. I was doing that all the time before I bought my current 18TB USB HDD which holds all the GOG installers, + other stuff. Before that I had two 5TB USB HDDs for my GOG installers (to which I divided my GOG games by dividing the manifest file into two), and even they were getting full.

Would such manual division be still possible with your suggested database approach?

I also use and edit the manifest myself and would appreceiate still being able to do so!

Since a database allows you to filter and sort its contents by any criteria you like it should be fairly trivial without even having to manually edit anything at all. For example, let's assume you have 1000 games. Just tell the program to sort the list of your games alphabetically and download the first 500 in directory a and the rest in directory b.

I usually let gogrepo download all games whichs slug I stored in a text file.
So the easiest way (for me) to do this would be

1. extract all slugs into a text file, can easily be done with grep
2. split the text file wherever wanted
3. download the games with the IDs stored in one of these text files to a specific directory.

Which program?

Re: Download Locations

A lot of folks have talked about splitting their manifest so as to change the download location. I'm assuming that's due to grabbing just about everything associated with their games (and not just because they have 100,000 games in their library :b )

For what it's worth I found it useful to set up a bunch of script (I run linux) files (batch files in windows) to download only what I want. And it changes directory as needed.

$PWD= "Present Working Directory" And for each set of downloads: And then I have similar scripts for mac, linux, and windows versions of the games. All going into different directories. So I just change into my gogrepo directory and run the script I want.

The only change I'll be making to those scripts is to split the log file into 3 (update.log, download.log, and verify.log) to make it easier to track errors.

Hope this helps some of you.

My program can do that. I'm programming my own downloader though it's been in alpha for quite some time. Such features are trivially easy to implement if you use a database for storing the data.

And so could gogrepo if it were refactored to use a database.

A DB is not needed for that. Working with lists works just as well.
It's just a couple of thousand entries max, very easy to handle.

I have 2399 games on GOG, downloading only the English Windows versions + extras. Those alone take something like 10 terabytes or even more already, so it isn't because I'd try to download all language versions and Linux + Mac too.

Also some extra space is needed for the orphaned (old) files which gogrepoc clean moves to a specific directory.

I earlier divided the downloads half and half to two 5TB USB HDDs. When it became harder and harder to fit them to those, I bought a 18TB USB HDD. I hope it will last for some time.

So, can you e.g. tell your script "download the first 800 games + extras to this path, then the next 800 games to path2, and the rest to path3", or how does it work?

Fundamentally, the scripts (batch files) that I have there just "pre-load" the command line for the downloader. For example in my "windows" version: The first 4 lines are just variables to make it easier for me to tweak (only have to change things in one place). The first one (GOGREPOC) is the base command - pwd is the linux command for "present working directory" - I use absolute paths. So the windows equivalent would be: python3 d:\gogrepo\gogrepoc.py Which assumes you're running the program from the directory "gogrepo" on your D:\ drive

LIBDIR is where I want the downloads to go.

Line 5 copies the token to the downloads directory
Line 6 changes directory into LIBDIR
Line 7 updates the manifest for the specific platform and language
Line 8 does the downloading
Line 9 verifies
Line 10 changes directory up 1 level (which I just noticed is a holdover from a previous version) so ignore it. It should go back to the original directory

Now according to Kalanyr's github, "savedir" is a command line switch that should accomplish the "where to put the downloads" piece. I haven't tested it yet.

To do what you are talking about (only the first 500 games, 501 to 1000, etc) you would either manually keep splitting the manifest, or modify gogrepoc.py to add 2 new command line options: Number to download and where to start.

There are over 100 forks of Kalanyr's program; and most look abandoned. Basically what you're looking to do here would be to put a few loops and checks into the program. You don't have to be a python expert to do that. Take the code and play with it. When you get it working the way you want, upload it either here or on github for Kalanyr to see if it should be added to the main branch.

Sorry if that last part come across as curt, it's not intended to be. I would do it myself, I just don't have the time at the moment

Lists are very useful and they do have the ability to sort and filter. But what I described above also requires a join and that's something lists can't do as far as I know. At least not in C#. I'm not really a snake charmer so I wouldn't know if Python can do that.

Well, lists themself neither filter nor join, but you can do all that with lists. Tables are also lists in a certain sense.
All that a DB offers are preimplemented functions to achieve your goal.
In a language that uses streams and functional operators these can be implemented quite readable as well
(keywords: removeIf, disJoint, retain,allMatch, noneMatch)

The difference is basically that for joining two "tables" (lists of objects with properties) and want "columns" from both, you need to create a new class for the result.
Since our game entries are entities with given properties, I don't see a reason to add more data fields to them through a join-Operation, but of course I might miss something here.
What would you join the table of games with in the first place?

Python has most of the list comprehensions that you can do in functional languages so list manipulation is actually vey powerful, it just shares the functional language requirement of being able to turn your brain into soup from a more traditional programming perspective to grock it.

Yeah I experimented earlier adding e.g. the estimation of remaining download time (which I think is still missing from the active versions of gogrepo?), but I got sidetracked and didn't feel like re-adding it to my personal version of the script every time Kalanyr's version is updated, and wasn't yet confident enough to upload it to github either...

Either way, splitting the manifest file before download is good enough solution for me, especially since I made a bash script that does that splitting automatically and copies the partial manifest files to different paths, I just run that script after gogrepoc update and before gogrepoc download (if needed). I don't need to touch the gogrepoc.py script myself.

Similarly I did some simple GOG installer verification script that can be used also without md5 checksums (as some installers seem to be missing them or there is some other problem with them; it uses innoextract to test the installers), but someone had already made a better and more comprehensive tool for that, GOG Download Checker or whatever it was called. I recall it also uses innoextract. It may be that tool was Windows-only though, while I tend to do this stuff on Linux (even though I download Windows installers), so I made a separate bash script for Linux, and a similar .bat file for Windows.

And so do lists. That's the beauty of it.

In C# for example lists have the methods Add, Insert, Remove, Find, FindAll, Exists, Sort, Reverse, just to name some of the most used ones.

If you want to read up on it see here.

Adding a complex operation like a join as a list method is of course possible but it is usually not a good idea to re-invent the wheel. Especially since most people won't be able to implement a complex operation like that as optimized as database server programmers can.

The table of files, obviously. Only the files table knows which files have not been downloaded and what type the file is (Installer, DLC, Patch, Extra) and what language (for games with separate installers for every language) while only the games table knows the name of the game. So without a join it is impossible to have a result table that is sorted by game names.