It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3896 posts)(3896 posts)
(3896 posts)
Pages:
1
50
100
150
200
213
214
215
216
217
260
This is my favourite topic
Posted July 02, 2022
avatar
SargonAelther: Ok so I gave gogrepoc by Kalanyr a try and it seems to work a bit better than gogrepo by eddie3.

Unfortunately I can't quite understand some of the syntax. IMO the ReadMe file is not good enough.

Can someone clarify the following:
1) How to specify the end of the "-lang" parameter. At the moment it seems to interpret every subsequent parameter as a language and then, naturally, fails. As a workaround I have to place the "-lang" parameter at the end. Are there any brackets or anything I can use?

2) How to use the "savedir" parameter. No matter whether I use quotation marks or not, whether they are single or double, it just fails. What is the proper syntax for the "savedir" parameter using a Windows path WITH spaces.

Every good Readme file should include examples for every command. Examples can often convey the information far better than any explanation.
Gogrepoc has integrated help (--help). Did you try it? I have never read the documentation. I always use it when I need something. There's both general help and command-specific help.

When I get back home, I'll post the command lines I use
Posted July 02, 2022
avatar
mrkgnao: Gogrepoc has integrated help (--help). Did you try it? I have never read the documentation. I always use it when I need something. There's both general help and command-specific help.

When I get back home, I'll post the command lines I use
I have checked the help option. It has significantly less information than the documentation. It only lists the primary commands, without any of the parameters.

The documentation at least provides the parameters for each command and a basic description, so I'll give the creators props for that, but the problem is that most of these do not have examples.

The "savedir" and "gamedir" parameters are also weird because they are listed without a "-" beforehand, unlike most other parameters, so I'm not even sure if it's a typo or what. The point is we need an example with a path for this, as I cannot understand the syntax of the path.
Post edited July 02, 2022 by SargonAelther
Posted July 02, 2022
avatar
mrkgnao: Gogrepoc has integrated help (--help). Did you try it? I have never read the documentation. I always use it when I need something. There's both general help and command-specific help.

When I get back home, I'll post the command lines I use
avatar
SargonAelther: I have checked the help option. It has significantly less information than the documentation. It only lists the primary commands, without any of the parameters.

The documentation at least provides the parameters for each command and a basic description, so I'll give the creators props for that, but the problem is that most of these do not have examples.

The "savedir" and "gamedir" parameters are also weird because they are listed without a "-" beforehand, unlike most other parameters, so I'm not even sure if it's a typo or what. The point is we need an example with a path for this, as I cannot understand the syntax of the path.
If it helps, these are the commands i use one at a time, where e:\gogbackups is my external drive dir destination:

@echo off

python gogrepoc.py update -os windows -lang en -full

exit

@echo off

python gogrepoc.py download e:\gogbackups

exit

@echo off

python gogrepoc.py clean e:\gogbackups

exit

@echo off

python gogrepoc.py verify e:\gogbackups

exit
Posted July 02, 2022
avatar
darkangelz: If it helps, these are the commands i use one at a time, where e:\gogbackups is my external drive dir destination:

@echo off

python gogrepoc.py update -os windows -lang en -full

exit

@echo off

python gogrepoc.py download e:\gogbackups

exit

@echo off

python gogrepoc.py clean e:\gogbackups

exit

@echo off

python gogrepoc.py verify e:\gogbackups

exit
Thank you! This is exactly what I meant by saying that examples convey information far better than any explanation!
Thanks to your example I now see that "savedir" was not a parameter that needs listing, only its value. Considering every other parameter needs to be listed, I would have never guessed this.

So now I used the following as a test and it worked:
gogrepoc.py download "G:\Data04\GOGRepoC Test" -id "18_wheels_of_steel_hard_truck" -os windows linux mac -lang en

Previously I was desperately trying some variation of:
gogrepoc.py download savedir "G:\Data04\GOGRepoC Test" -id "18_wheels_of_steel_hard_truck" -os windows linux mac -lang en

or

gogrepoc.py download -savedir 'G:\Data04\GOGRepoC Test' -id "18_wheels_of_steel_hard_truck" -os windows linux mac -lang en

etc. and none of them worked.



On a separate note, is there any way to estimate how much space would the entire library take once downloaded?
Post edited July 02, 2022 by SargonAelther
Posted July 03, 2022
avatar
mrkgnao: Gogrepoc has integrated help (--help). Did you try it? I have never read the documentation. I always use it when I need something. There's both general help and command-specific help.

When I get back home, I'll post the command lines I use
avatar
SargonAelther: I have checked the help option. It has significantly less information than the documentation. It only lists the primary commands, without any of the parameters.
You can get detailed help with all the parameters by requesting command-specific help (e.g. gogrepoc update --help or gogrepoc download --help).

I guess you managed to figure things out with the help of darkangelz examples, so I expect there's no need for my posting my command lines any more. Right?

Good luck and remember to do full updates often.
Posted July 03, 2022
avatar
SargonAelther: On a separate note, is there any way to estimate how much space would the entire library take once downloaded?
i never tried it, but i've seen it mentioned that's what -dryrun is for. Not sure of the syntax tough, If something like this would be enough, or still needs the dir destination in there:


@echo off

python gogrepoc.py download -dryrun

exit
Posted July 03, 2022
For the documentation, my apologies, I was in the process of rewriting stuff to use the new APIs and add some functionality (like automatic updates) when I get mugged by other projects and then plague, so it doesn't really seem worthwhile to update the old documentation (and as it currently stands if I had the time I'd rather focus on finishing that rewrite and then documenting that)

avatar
blotunga: I had lots of issues downloading Imperator Rome (both with the browser and gogrepo), I wonder what happened there. I finally managed to grab the .bin file via the browser but after countless retries.
It's borked on GOG's end, and in a different state on pretty much every server they use, there's like ~1 server that has all the files. I have a separate backup folder for Imperator and update it on the 1/6 month occasion I hit the right server if anything has changed.

I badgered them into fixing it once before ,but it desycnhed with the next change they did. >_>
Post edited July 03, 2022 by Kalanyr
Posted July 04, 2022
avatar
MMLN: Well I have that issue when using this command

"gogrepoc.py update -os windows linux -lang en -full"

and it is happening to all of my games, not only to AoW :(

11:56:55 | ( 18 / 258) fetching game details for beneath_a_steel_sky...
11:56:55 | The handled exception was:
11:56:55 | error
Traceback (most recent call last):
File "D:\Backup\GOG\gogrepoc.py", line 1379, in cmd_update
item_json_data = response.json()
File "C:\Program Files\Python39\lib\site-packages\requests\models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Program Files\Python39\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Program Files\Python39\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\Python39\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
11:56:55 | End exception report.

EDIT: OK disregard it I have found the issue. The stored cookie from my last session has become for some reason invalid. Running the "gogrepoc.py login" command again solved my issue, and it now updates correctly...

EDIT2: If Kalanyr reads it, maybe there would be nice to have some error message explaining that a person needs to login again, if this error message pops up all around your screen, to be less confusing, for the noob users, like myself :)
avatar
Kalanyr: There should already be detection for if you're logged out which notifies you that you need to log in again but I guess GOG has changed something that's broken it. I'll look into it at some point but I don't think it's a super high priority and I'm working on something else eight now.
No worries, I have found where the issue was after few try and error runs :D Unfortunately the notification did not trigger at all, therefore I was little bit confused, but in the end, your program saves me a lot of hassle each time, I decide to update offline backup of my catalogue :)
Posted July 04, 2022
avatar
Kalanyr: For the documentation, my apologies, I was in the process of rewriting stuff to use the new APIs and add some functionality (like automatic updates) when I get mugged by other projects and then plague, so it doesn't really seem worthwhile to update the old documentation (and as it currently stands if I had the time I'd rather focus on finishing that rewrite and then documenting that)

It's borked on GOG's end, and in a different state on pretty much every server they use, there's like ~1 server that has all the files. I have a separate backup folder for Imperator and update it on the 1/6 month occasion I hit the right server if anything has changed.

I badgered them into fixing it once before ,but it desycnhed with the next change they did. >_>
Ah ok, thanks, good to know. I always check the orphaned files. For example at one point Master of Magic Caster of Magic for Windows was in the library for everyone who owned Master of Magic...
Posted July 05, 2022
avatar
Kalanyr: For the documentation, my apologies, I was in the process of rewriting stuff to use the new APIs and add some functionality (like automatic updates) when I get mugged by other projects and then plague, so it doesn't really seem worthwhile to update the old documentation (and as it currently stands if I had the time I'd rather focus on finishing that rewrite and then documenting that)
The downside of inheriting an existing project is that sometimes, you don't have as much control as you'd like over the tooling.

Writing a decent command line interface from scratch is a significant amount of work. If you can take advantage of existing third-party libraries that do most of that heavy lifting for you, it tends to be a good idea to do so (whatever effort you make writing things from scratch on you own with the limited time you have is almost guaranteed to be inferior to what the library provides).

Most very popular languages have at least one well developed command-line library that can do most of the heavy lifting for you in terms of providing a rich command line interface that is self-documenting with help commands.

For golang, I use Cobra. In Python, you have the click library which I used in the past and is really awesome: https://click.palletsprojects.com/en/8.1.x/

If you're re-writing stuff anyways, you might consider integrating the above in your tool and take advantage of its rich functionality.
Post edited July 05, 2022 by Magnitus
Posted July 05, 2022
I have noticed that GOG has some naming issues with some games. For example Mac Versions of Syberia 1 and 2 use the same exact file name for every language. So both English and French are called syberia_en_2.0.0.1.pkg.
Galaxy overwrites these with the last downloaded version for example, so if you wanna keep all versions you have to manually rename them.

I will test to see how GOGREPO handles this, but this is why subfolders should be used for things like extras, patches and languages. In the case of patches, I wish there was a way to skip them entirely.

avatar
mrkgnao: Good luck and remember to do full updates often.
That's actually a good advice. The "regular" update seems too only query GOG servers for the "Updated" tag, aka the blue dot. Any new DLC additions get overlooked, while a full scan does find them. It's a big annoying that some game information still fails to be acquired fully and it can be a bit hard to make a clear log of what's missing.

avatar
darkangelz: python gogrepoc.py download -dryrun
Yeah, it does mention the size at the end, thanks.
Post edited July 05, 2022 by SargonAelther
Posted July 05, 2022
avatar
SargonAelther: I have noticed that GOG has some naming issues with some games. For example Mac Versions of Syberia 1 and 2 use the same exact file name for every language. So both English and French are called syberia_en_2.0.0.1.pkg.
Galaxy overwrites these with the last downloaded version for example, so if you wanna keep all versions you have to manually rename them.

I will test to see how GOGREPO handles this, but this is why subfolders should be used for things like extras, patches and languages. In the case of patches, I wish there was a way to skip them entirely.
I believe both my client and gogrepoc handle those correctly.

My client numbers them with an increment within square brackets. As someone some time ago asked me if I could change it to regular backets to behave like gogrepoc, I believe gogrepoc does the same, but with regular brackets.

However, be aware that sometimes, it is the exact same file across languages (yes, some installers containing all languages in one installer are duplicated for each language in the listing). gogcli eliminate those duplicates by comparing the checksum (if it is the same, it considers it the same file). Not sure how gogrepoc behaves in this regard.
Post edited July 05, 2022 by Magnitus
Posted July 05, 2022
Please do not do full updates more than once a month or so or during Sales/Events. If you know (a) specific game(s) has/have been updated or DLC has been added then use the -ids option to specifically update that/those game(s). There's a reason it doesn't default to the full update and that's because full updates hammer the server pretty hard.
Posted July 05, 2022
avatar
Kalanyr: Please do not do full updates more than once a month or so or during Sales/Events. If you know (a) specific game(s) has/have been updated or DLC has been added then use the -ids option to specifically update that/those game(s). There's a reason it doesn't default to the full update and that's because full updates hammer the server pretty hard.
That's fine. What about the Download function though. What does it do when some games are already present? Will it skip them, or will it redownload everything again?

Also I noticed that the download speeds are only about 2MB/s. Actually suspiciously around 2MB/s. Fluctuates between 1.9 and 2.1. Using a browser or Galaxy I can easily get 30+MB/s. Is something throttling it?

My library is about 10TB, so downloading it at 2MB/s will take me over a hundred days.
Post edited July 05, 2022 by SargonAelther
Posted July 05, 2022
avatar
SargonAelther: Also I noticed that the download speeds are only about 2MB/s. Actually suspiciously around 2MB/s. Fluctuates between 1.9 and 2.1. Using a browser or Galaxy I can easily get 30+MB/s. Is something throttling it?

My library is about 10TB, so downloading it at 2MB/s will take me over a hundred days.
Sometimes, GOG's servers/CDN is under heavy load and/or slower (Timboli had that issue for a while and it eventually resolved itself).

A good rule of thumb is to compare the performance with what you get when you download a file from the browser and that will give you an idea. In the case of Timboli, if I recall correctly, it was slow in the browser too.

Otherwise, my connection is ~6 megabytes per second (50 megabits per second) and it tends to be the download speed that I get when my home network is not saturated with other traffic.
Post edited July 05, 2022 by Magnitus
Pages:
1
50
100
150
200
213
214
215
216
217
260
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3896 posts)(3896 posts)
(3896 posts)