It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3849 posts)(3849 posts)
(3849 posts)
Pages:
1
50
55
56
57
58
59
150
200
250
257
This is my favourite topic
Posted July 22, 2017
avatar
Kalanyr: Apologies. The build discussed above didn't upload for some reason. It's uploaded now.
Should we rebuild Manifest after moving to your fork?
Posted July 23, 2017
avatar
Kalanyr: Apologies. The build discussed above didn't upload for some reason. It's uploaded now.
avatar
Starkrun: Should we rebuild Manifest after moving to your fork?
Shouldn't be necessary now, I think I've fixed all.the points where the non-Gakaxy including layout would cause problems. Obviously the skipshared and skipgalaxy flags won't work on the old entries though and skipstandalone will skip what would be both standalone and shared for those entries.

I recommend doing a full update when you can though (there's no need to delete and rebuild from scratch or anything a full update will detect anything in your manifest no longer on gog and remove it, so a full update and creating from scratch should end exactly the same).
Posted July 23, 2017
avatar
Starkrun: Should we rebuild Manifest after moving to your fork?
avatar
Kalanyr: Shouldn't be necessary now, I think I've fixed all.the points where the non-Gakaxy including layout would cause problems. Obviously the skipshared and skipgalaxy flags won't work on the old entries though and skipstandalone will skip what would be both standalone and shared for those entries.

I recommend doing a full update when you can though (there's no need to delete and rebuild from scratch or anything a full update will detect anything in your manifest no longer on gog and remove it, so a full update and creating from scratch should end exactly the same).
Thanks for the replay i'll append a backup on my current manifest and just generate overnight... Kyn, walking dead season 1, Dreamfall Chapters, gwent, and x3 terran always gave me issues on one part or another... curious to see how this works.. nice to see you choose classic installers as default so i don't have to go messing up my old ps1 scripts :)

Thank you for keeping this alive, I know woolymethodman has had a lot of real life happening for him. I think I speak for many people when I say, we all truly appreciate your work on this!
Posted July 23, 2017
avatar
Starkrun: Thanks for the replay i'll append a backup on my current manifest and just generate overnight... Kyn, walking dead season 1, Dreamfall Chapters, gwent, and x3 terran always gave me issues on one part or another...
Aren't those the games which either miss the md5 checksum, or give the xml parsing error? I think the problem is at GOG's end. For us it just means that those particular games will not be verified against their md5 checksum (as they have none available) by gogrepo, but only their size is checked. They still get downloaded fine, though.

These are the problems I get, every time (for many many months already, maybe even a year) when running the full update. Or do you get some other kinds of errors? I don't get any error with Dreamfall Chapters (Season Pass) though.

00:59:24 | ( 8 / 1372) fetching game details for a_golden_wake...
00:59:31 | xml parsing error occurred trying to get md5 data for setup_a_golden_wake_2.0.0.4.exe
...
04:03:55 | ( 602 / 1372) fetching game details for kyn...
04:04:02 | request failed: HTTP Error 404: Not Found. will not retry.
04:04:02 | no md5 data found for setup_kyn_2.1.0.4.exe
04:04:07 | request failed: HTTP Error 404: Not Found. will not retry.
04:04:07 | no md5 data found for setup_kyn_2.1.0.4-1.bin
...
07:35:42 | (1297 / 1372) fetching game details for walking_dead_season_1_the...
07:35:58 | request failed: HTTP Error 404: Not Found. will not retry.
07:35:58 | no md5 data found for patch_walking_dead_season_1_2.1.0.5.exe
...
07:50:04 | (1343 / 1372) fetching game details for x3_terran_war_pack...
07:50:31 | xml parsing error occurred trying to get md5 data for patch_x3_terran_war_pack_2.2.0.5.exe
EDIT: Oh, and for The Walking Dead and X3 Terran, the missing md5 checksum seems to be only for patches, so I couldn't care less for those two... The main installers apparently still have the md5 checksums. Only Kyn and A Golden Wake miss the md5 checksum for the main installers, in my case.
Post edited July 23, 2017 by timppu
Posted July 23, 2017
Now that I found the way to divide my gogrepo installer collection into two, that also speeds up my downloads and installer verification greatly.

Because I have two unused 2TB USB hard drives, but my GOG installer + extras collection (Windows/English only) is already bigger than 2TB, I've divided the GOG collection to those two drives so that GOG games which start with A-N (or numerals, like 1nsane, 7th Legion and so on) are on one USB drive, and O-Z are on the other.

After I've done a full update getting a full manifest file, I then make two copies of it and from one remove all the titles from #-N, and the other I remove O-Z. Using these two separate partial manifest files, I can run gogrepo download (and verification) on two separate PCs (laptops) at the same time, using separate internet connections. I have two internet connections I can utilize, my main 10Mbps cable modem connection, and my unlimited 4G mobile connection that gives me around 40Mbps at my home on a good day (It has reached around 120Mbps or so elsewhere, but not at my home unfortunately).

So right now I am downloading the new and updated #-N installers using the 10Mbps cable modem connection, and the O-Z installers with the 40Mbps mobile connection. GOG has changed lots of installers lately so for the first patch I have something like 75 GB to download, and for the second batch a whopping 200 GB to download. Very nice that I can now download them using a total 50Mbps speed, utilizing all my available bandwidth.

Earlier I did try some internet suggestions on how to merge two different internet connections into one in Windows (to get benefit of both bandwidths on the same device), but for some reason I never got them to work, I'd still get the bandwidth only from one of them.

Naturally also verification will speed up as now I can verify #-N and O-Z at the same time, running gogrepo twice at the same time.
Post edited July 23, 2017 by timppu
Posted July 29, 2017
avatar
timppu: After I've done a full update getting a full manifest file, I then make two copies of it and from one remove all the titles from #-N, and the other I remove O-Z. Using these two separate partial manifest files
Perfect application for wild cards in the -id option. Using real regular expressions you could use -id ^[0-9a-n]* and -id ^[o-z] with the same manifest file with two commands (I'll assume the manifest is open r/o for download commands). I'm working on this ...
Posted July 29, 2017
avatar
sbaylus: Perfect application for wild cards in the -id option. Using real regular expressions you could use -id ^[0-9a-n]* and -id ^[o-z] with the same manifest file with two commands (I'll assume the manifest is open r/o for download commands). I'm working on this ...
Yeah that would certainly be nice as that would drop the need for little manual work with the full manifest file (dividing it into two).
Posted July 31, 2017
Ever since the Galaxy installers, I did not update any more :*(
@Kalanyr, I appreciate the work. Will check this out tomorrow, I think.
Posted August 01, 2017
avatar
disi: Ever since the Galaxy installers, I did not update any more :*(
Why not? gogrepo downloads the classic installers by default, not the Galaxy-embedded installers. So you are as good to go as ever.

Be aware though that lots of classic installers have been updated recently as well. 1½ weeks ago I redownloaded around 270GB of installer files (most were those updated ones, but I had also bought some big new games), but this week there hasn't been similar flood of updated installers (I got only a few updated ones last week).

I just updated/downloaded my collection yesterday, went fine. I noticed though there was the same "two files with the same filename" problem as people have reported in this thread before with Sudden Strike 3, but today it was fixed. The glitch caused gogrepo to repeatedly redownload the said installer whenever one runs gogrepo download (only once per command).

Which reminds me:

avatar
timppu: Ok so for Michonne only the first three installer files are needed, and for Reigns only the first one? And we don't know why there are these duplicate files?

At least the download pages for both games are wrong then, as they suggest Michonne is in six parts (three parts in reality), and Reigns in two parts (when it is only in one). Yeah GOG needs to fix those download pages either way,
avatar
Kalanyr: Michonne and Reigns have different but similar problems.

Michonne is as you said.

Reigns has 64 and 32 bit versions , but the 32 bit version is present as both Part 2 and as a separate download (both with same name but different size / md5) , whereas Part 1 is the 64 bit version.

Unfortunately I have no idea which of the "duplicates" is the correct one. While GOG Galaxy exists it doesn't matter because it will repair the wrong one automatically by updating it. So yeah you only need 1 complete set of them but I have no idea which one that should be.
Have these games (different files with identical filenames on the same download page) been reported to GOG, by the way?

I noticed the same problem with the "new" release of Sudden Strike 3: Arms for Victory, ie. it had two installer files (1 or 2, 2 of 2) which had identical filenames, causing gogrepo to redownload the installer whenever you run the gogrepo download command. I reported it to GOG support, and they fixed it within a day (now there is only one installer file).

I can't check whether those aforementioned games are fixed too as I don't have them.
Post edited August 01, 2017 by timppu
Posted August 01, 2017
avatar
disi: Ever since the Galaxy installers, I did not update any more :*(
avatar
timppu: Why not? gogrepo downloads the classic installers by default, not the Galaxy-embedded installers. So you are as good to go as ever.

Be aware though that lots of classic installers have been updated recently as well. 1½ weeks ago I redownloaded around 270GB of installer files (most were those updated ones, but I had also bought some big new games), but this week there hasn't been similar flood of updated installers (I got only a few updated ones last week).

I just updated/downloaded my collection yesterday, went fine. I noticed though there was the same "two files with the same filename" problem as people have reported in this thread before with Sudden Strike 3, but today it was fixed. The glitch caused gogrepo to repeatedly redownload the said installer whenever one runs gogrepo download (only once per command).

Which reminds me:

avatar
Kalanyr: Michonne and Reigns have different but similar problems.

Michonne is as you said.

Reigns has 64 and 32 bit versions , but the 32 bit version is present as both Part 2 and as a separate download (both with same name but different size / md5) , whereas Part 1 is the 64 bit version.

Unfortunately I have no idea which of the "duplicates" is the correct one. While GOG Galaxy exists it doesn't matter because it will repair the wrong one automatically by updating it. So yeah you only need 1 complete set of them but I have no idea which one that should be.
avatar
timppu: Have these games (different files with identical filenames on the same download page) been reported to GOG, by the way?

I noticed the same problem with the "new" release of Sudden Strike 3: Arms for Victory, ie. it had two installer files (1 or 2, 2 of 2) which had identical filenames, causing gogrepo to redownload the installer whenever you run the gogrepo download command. I reported it to GOG support, and they fixed it within a day (now there is only one installer file).

I can't check whether those aforementioned games are fixed too as I don't have them.
Yeah,.they were fixed as of about 2 weeks ago.
Posted August 18, 2017
When movie support is added, I hope that excluding films is easy. I have a couple of (free) 10 GB films on my account and I'd prefer not to have them on my hard drive, in addition to not straining GOG servers.
Post edited August 18, 2017 by African_wildlife
Posted August 19, 2017
I tried to see if I could login with gogrepo if my cookies file were gone.
I disabled two-step authentication in my gog.com account.
I am getting this message now in gogrepo:
"cannot continue, gog is asking for a reCAPTCHA :( try again in a few minutes."

EDIT: ok, I waited roughly 15 minutes. I can now login with gogrepo.
Post edited August 19, 2017 by solar_dome
Posted August 19, 2017
I am now getting these errors:

python3 gogrepo.py update -os windows linux mac -lang en
23:35:53 | loading local manifest...
23:35:53 | fetching game product data (page 1)...
23:35:54 | fetching game product data (page 2 / 2)...
23:35:54 | found 150 games !!!
23:35:54 | ( 1 / 150) fetching game details for 1nsane...
23:36:05 | error
Traceback (most recent call last):
File "gogrepo.py", line 928, in cmd_update
item.downloads = deDuplicateList(item.downloads,{})
File "gogrepo.py", line 454, in deDuplicateList
dummy_item = copy.copy(update_item)
File "/usr/lib/python3.4/copy.py", line 97, in copy
rv = reductor(2)
File "gogrepo.py", line 199, in __getattr__
return self[key]
KeyError: '__getstate__'
If I let it continue, possibly more of the same.
Post edited August 19, 2017 by solar_dome
Posted August 19, 2017
avatar
solar_dome: I am now getting these errors:
Just a shot in the dark: do you already have an old manifest file? Since you are apparently trying to get all file details all over again (not using -skipknown or -updateonly options), delete any old manifest file.

I always redownload the whole manifest file all over again just so that I am guaranteed to catch any changes, so I always also delete the old manifest file before running gogrepo update.

Also, are you running the original gogrepo, or one of the recent forks by different authors? For now I am just using the vanilla gogrepo.
Post edited August 19, 2017 by timppu
Posted August 20, 2017
avatar
solar_dome: I am now getting these errors:
avatar
timppu: Just a shot in the dark: do you already have an old manifest file? Since you are apparently trying to get all file details all over again (not using -skipknown or -updateonly options), delete any old manifest file.

I always redownload the whole manifest file all over again just so that I am guaranteed to catch any changes, so I always also delete the old manifest file before running gogrepo update.

Also, are you running the original gogrepo, or one of the recent forks by different authors? For now I am just using the vanilla gogrepo.
I deleted my old manifest file before I ran gogrepo, and received the errors.

I am running gogrepo from Kalanyr.

gogrepo functioned successfully using python2.7.
python2.7 gogrepo.py update -os windows linux mac -lang es ko sk cn ro sb hu fr gk sv jp it ar bl nl pt pl fi tr en cz ru no da de br
Post edited August 20, 2017 by solar_dome
Pages:
1
50
55
56
57
58
59
150
200
250
257
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3849 posts)(3849 posts)
(3849 posts)