I am personally using the dev-branch version (gogrepoc-dev), apparently it has more fixes and stuff and is more up to date. I haven't checked the exact changes.

I presume "games" is a subfolder under gogrepo-master, under which all your GOG games are? Yes I think you should use the clean command too, otherwise your game installer subfolders will grow and grow, keeping also all the old files there. "verify -delete" will not remove those obsolete files, as it only checks files it recognizes (the latest files), and deletes them only if they fail the integrity check.

Also unless you tend to remove your manifest file (in which case the update command will create it from scratch), I am unsure if "-updateonly" and "-skipknown" really matter nowadays (it has always been a bit unclear to me what their EXACT purpose is, in the big picture). To me it seems gogrepo will skip most of the files on your existing manifest file anyway, not sure what purpose those options have anymore (kalanyr can explain I guess).

Here is an example of my gog.bat file which makes sure I have a updated, clean and verified set of GOG game installers (English Windows versions only; as said I am using the dev version, gogrepoc.py):

python gogrepoc.py update -lang en -os windows
python gogrepoc.py clean e:\gog\
python gogrepoc.py download e:\gog\
python gogrepoc.py verify e:\gog\

e:\gog\ is the subdirectory of the external 5TB USB hard drive (always E: drive for me) where my GOG game installers are.

The clean command moves any old and obsolete files under the !orphaned directory, before I even download. I guess it doesn't matter much if you do clean before or after downloading new files. Having it before downloading new files, at least if you notice that you don't have enough space to download all your new and changed games, you have time to manually delete any obsolete files that you don't need, before running out of space.

I don't use the delete option with verify, I just check from the report which files failed verification and check them manually, and remove if necessary. I guess if you want it to be as automated as possible, then yes verify .delete and then run download again (to get any deleted files again).

You just need to occasionally remember to check !orphaned directory if you want to delete old files from there (or keep them), but if you feel certain you won't ever miss any old files, I guess you can delete all the files inside !orphaned (just to make sure that directory doesn't grow uncontrollably, if you are running out of free space).

If for some reason you need to exit the process (whether it is during update, download or verify; clean doesn't last many seconds anyway as it quickly checks and moves files around), you can exit it with Ctrl-C. Then just run your bat file again, it will quickly go back to the poiint where you left. It will run update, download and verify again, but skip all the things quite fast which were already done before. You'll see when you try it out. :)

Affirmative. :)

What gogrepo still seems to be missing (and lgogdownloader too) is an estimated remaining download time for all your files. I was earlier checking a good solution for that with python in order to propose it to kalanyr (I found some quite good examples how to do it on python), but I couldn't quite figure out how the total download speed could be calculated in gogrepo currently, or maybe it was something else... then I got distracted.

For people who e.g. run gogrepo the first time, I think it would give quite good feedback, some indication whether downloading all the games will take hours, days, weeks, MONTHS...? I recall when I originally did that, I had absolutely no idea how long it would take (without doing some calculation manually), it would have been nice to have SOME estimation, without having to calculate it manually.

Yes yes download speeds fluctuate, but the algorithms calculating the remaining time try to take that into account (I found one simple algorithm which seemed quite effective, yet simple). Also, at least in my case, the speed doesn't really fluctuate that much. I have a 10Mbps cable modem (which tops out at around 1.2 Mbytes/sec normally), and when using e.g. gogrepo, it doesn't really seem to fluctuate that much. It stays at about 1.1 - 1-2 Mbytes/sec almost all the time during the whole download.

Is that dev branch safe? Kalanyr didn't release it as he wasn't yet sure about stability..

Btw where is he? The last change in prod was in August and the last one in dev was in September.
I hope that it won't happen like with Woolymethodman, who disappeared right before releasing a major version.. O_o'

I recall that someone implemented "skipfiles" and Kalanyr added it to his dev version.
I would have used it like this (but it never reached prod):
-skipfiles 'patch_*' '*hotfix*' 'setup_*_32_*'

Seems to work just fine for me, frankly I don't see much difference. update, clean, download, verify, same same. What is the worst that could happen anyway with a python script?

It seems people who know python are always otherwise busy too, The original author seemed to be often very busy with work and real life, and to me kalanyr has appeared the same.

Then again, anyone can make their own fork out of gogrepo and start contributing. Python is one of the "sanest" programming languages to learn for beginners, so go ahead!

I know some python but gave up when I couldn't easily figure out how to get the combined download speed out of current gogrepo (in order to calculate the total remaining time with a cool simple algorithm I found online). :D

Things happen with people's lives, relocate, get a new job, get unemployed, get married, get kids, get divorced, die, get resurrected etc. Many reasons why maintaining some cool tool is not always on the top on the mind.

Glad to know that!
And that!

For any new comers here is the quick and dirty:

Use python 3 for gogrepo.py

main branch at https://github.com/Kalanyr/gogrepo
dev branch at: https://github.com/Kalanyr/gogrepoc/tree/dev

Requirements that need installed and updated:
-html5lib
-html2text
-six
-requests
-pyopenssl

Use "pip install six" as an example to install these in python.

I initially used the "update" command with more than one language and now want to only keep English (I think I will download other languages manually on a case-by-case basis). It doesn’t seem possible to directly update the manifest file, should I just delete the manifest file and use "update" again ? Will this break updates for the games I have already downloaded ?

And I can confirm that Kalanyr’s master branch only downloads Windows versions for me too ; I have to use the dev branch

What version is this?

Holy hell, i don't know what's worse: trying to follow the current version or trying to follow the features without docs or trying to follow the gog/gogrepo bugs. I feel like we could benefit from a bugzilla and just archive this thread to point to that instead. I feel like those of us in the thread, Kalanyr included, are just writing the documentation over and over again for those new to the project or those who fell out of the loop long enough. All i know right now is, i'm probably behind to some degree, and I can't write my update to my script that i run gogrepo in until i update, but some of the issues make me afraid to update, for fear that most of my files will be gone due to a bug or something, since i know the manifest will have to be rebuilt.

I feel like if we went ahead and just built a littlebit of something resembling tools for this information, we could probably reduce the workload on Kalanyr, not only by just having less to talk about but by also giving him more freedom since the documentation will come naturally or something. The project clearly has a community of it's own subordinate to gog, but it's not being utilized effectively. Just like how GOG can't host the source files, i feel like we should offload the discussion elsewhere and see what we can do about getting this topic (or another pointing to the right site) stickied with a disclaimer so GOG's not held responsible.

Kalanyr, if you reading this, and you want to get the ball rolling on this, I could probably get us an IRC room or something. I have my own IRC server (not sure what kind of load it can handle, yet, as i'm still trying to maintain the logbot, and some other private features I came up with to help use IRC to replace Skype [since that's just getting more and more unstable]), and i'm sure Freenode or another similar group (since we all have webchat features) would be able to host. This could allow people with a little more time to help organize things. It's just, since you're the dev of the project, we can't really do anything and call it official without your say-so, and GOG's forum seems like more work than offloading this to something else. gogrepo needs a forum at the very least, not a thread (especially because you're announcing updates in a tech support area, essentially). PM me if you agree, and i can start trying things out and sending the suggestions to you. I'm sure with some sort of community effort you could offload the documentation to a volunteer, and even have some community maintained instructions for getting everything working.

EDIT: I have xxx, if you don't want to add me as a friend here on gog. Obviously, if you choose this route, please register your nick (use a fake gmail or something: "/msg nickserv help" for more information), and then tell me here (for confirmation) what you registered as, as i can't just assume someone isn't trolling me by coming on as Kalanyr. As the name temp implies, this is more or less for a more realtime communication for a more (semi-)permanent solution (especially with forum drama and the like, it's probably best if we ride on something a little more independent of the forums more long term, which is also good because some people could experience issues registering with nickserv if they have an overzealous email provider).

Just pinging you.

Until it gets pushed off the page and buried in obscurity. Which won't be long.

Does the script have some way to tell, how much space will all the downloaded stuff take? (I'd probably need to buy a new hard drive for the games, but would need to know, how large.)

Yes, it does, though it doesn't currently feature a way for you to query that information directly other than starting the download.
When the download starts, it shows you a running tally of how much has been downloaded and how much is left.

If you know a little Python, or other programming language, you could easily filter the manifest for all lines starting with "size" (after the whitespace) and then accumulate all the sizes.
Are you on Linux? If so, this one-liner should do it:

cat gog-manifest.dat | grep "^ *'size': [0-9]\+" | sed -e "s/ *'size': \([0-9]*\),/\1/g" | paste -sd+ - | bc

Run the script with -dryrun. It will produce a list of what it will download with the total size without actually downloading.

For those still having trouble with the login, as well as in case of future issues, I'm going to be making a cookie grabber that will work universally with every browser (without plugins, too). In order to make it so Kalanyr optionally can import the code directly into gogrepo if he chooses, i'm going to be learning python. Apparently the task is trivial in python, but it requires some very out of the box thinking. I'll post screenshots when i get it working, obviously using cookies from my own website in a worthless subdirectory for safety.

And, of course, it'll have utility outside of just GOG.

If the moderators feel uncomfortable with me posting this, feel free to message me and I'll promptly purge it from my server. Honestly, this isn't really hacking, though, and presumably this would require the gog user's consent. This would definitely show the importance of using HTTPS, however, as it seems to break (i'm not exactly sure how it responds differently) when you HTTPS. I did test, and if I attempt to connect to gog using HTTP, it will work, and i did see some cookies.

Special cookie page

Source code for special cookie page

Cookie page compromised warning

Exposed cookies

Python source code

So what's going on here is I'm using the python script as an HTTP proxy, which then means everything has to go through it. However, since all we want is the cookies for use with gogrepo.py, that's all I end up returning. I only tested with Firefox, and out of date at that, but I imagine it'd work either way. The reason this works is that cookies are only meant to be available to the same domain that created them (and sometimes only for a specific directory on that site), so the browser filters out cookies from other sites and then sends them as part of the HTTP request to the site. This all must go through the proxy, though an alternative method, although easy to extrapolate from what i wrote, would very likely be used as a hacking tool, so I'm not going to discuss it. Requiring the user to use this as a proxy thus shows consent.

And to people reading this who don't understand, don't let anyone else run the script for you. You should only ever run it for yourself, otherwise you're giving people your cookies which would give them access to your account. Oh, and for the love of God, disable the proxy settings when you're done, otherwise you'll find it hard to connect to the website again (unless you use HTTPS, but then regular HTTP sites will not work, which is a good number of them).

pip is a package manager for python. It's in the same directory as python, and is not meant to be used by files that run in python.

EDIT: I just realized i derped pretty hard core. I'm just going to leave the derp there, 'cause that's the type of mood i'm in.

Hey, I have a question : the external drive I’m now using for backing up the games is formatted with NTFS, which apparently doesn’t support posix_fallocate and in that case, it seems that posix_fallocate writes zeroes to each sector that we want to allocate. I see that it is indeed very slow, but it’s not really the speed that concerns me… isn’t that physically bad for the drive ?