It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4246 posts)(4246 posts)
(4246 posts)
Pages:
1
50
100
108
109
110
111
112
200
250
284
This is my favourite topic
Posted August 11, 2018
avatar
kohlrak: Supposedly, they got rid of those, though, remember?
avatar
Kalanyr: No ? I keep them in my manifest for testing purposes and they've never been removed. GOGrepo update just doesn't get them by default since they aren't much use to most people.
Then how would people get them? That's weird, seeing as "bundled installers" right now only include galaxy and a command to install galaxy. GOG might benefit from deleting those installer and not only saving space but presumably alot of other things would run more stably, so maybe we'd stop seeing as many issues when using gogrepo.
Posted August 11, 2018
avatar
Kalanyr: No ? I keep them in my manifest for testing purposes and they've never been removed. GOGrepo update just doesn't get them by default since they aren't much use to most people.
avatar
kohlrak: Then how would people get them? That's weird, seeing as "bundled installers" right now only include galaxy and a command to install galaxy. GOG might benefit from deleting those installer and not only saving space but presumably alot of other things would run more stably, so maybe we'd stop seeing as many issues when using gogrepo.
They are the default option from the GOG.com website still, and you can control what gogrepo gets on update using the installers option (which can be standalone, galaxy or both) , I use both for update but only download the standalone ones , it's so I know what's going on if someone is having issues with galaxy downloads. They are just standard game installers that also include a Galaxy stub last I checked, so yeah they are pretty useless but *shrug*.
Post edited August 11, 2018 by Kalanyr
Posted August 11, 2018
avatar
Kalanyr: Hmmm. GOG has changed the logic of the login page slightly which broke the old way of getting the login URL. A quick fix is easy.

Change:
login_data['auth_url'] = elm.text.split("'")[1]
to
login_data['auth_url'] = elm.text.split("'")[3]
Thanks!
Posted August 11, 2018
avatar
Starkrun: whats up with the files that have the (g) in their title?
avatar
Kalanyr: Probably installers with the Galaxy stub built in.
hand parsed them all, almost everyting was galaxy installers, files removed by gog, or missing md5... found a bunch of little mistakes too i'll write up and send them to gog.

sidenote, I'm excited again about scripting/programing... summer upgrades are over and school starts in a week. Im going to have much more time to mess around and read now. Im the end level breakfix for my company, they send me out when the rest fail so summer is when we strap on a backpack, grab our tools and run for 2 months straight.

My first foray into py3 was yesterday and im loving its integration into windows bash (win 10) going to see if i cant convert some powershell scipts to python for fun and to learn more ;) again thank you for all the help!
Posted August 11, 2018
Just made a commit that makes the authentication url detection a bit more robust, it will also give a nicer error message and break cleanly if it can't find the authentication url.

I've backported this to master since it's kind of functionality breaking.
Posted August 12, 2018
I've been tinkering with it a little bit to add a couple of changes that were worthwhile for me :

- A new parameter -skippatches that skips downloading the patches (patches meaning files starting with patch% for the time being)
Ideally we would like to download the installer and any patch more up-to-date that that.

- A new parameter -onelanguage that downloads the files just for one language in -lang order. In my -lang es en case meaning that will only download the spanish installer. If there isn't the it would download the english installer.

(both very straighforward additions, as I don't know python at all).

Instead of:
downloadsOS = [game_item for game_item in item.galaxyDownloads if game_item.os_type in os_list]
I changed it to:
https://pastebin.com/BpLsfAEf
(for all three downloads, galaxyDownloads and sharedDownloads)

And instead of:
downloadslangs = [game_item for game_item in item.downloads if game_item.lang in valid_langs]
I changed it to:
https://pastebin.com/kst5Yy98
(also for all three downloads, galaxyDownloads and sharedDownloads)

PS: Obviously there are also changes to add the new parameters to the procedures and to show de proper info of the new parameters in the download -h call.
Post edited August 12, 2018 by rgnrk
Posted August 12, 2018
avatar
rgnrk: I've been tinkering with it a little bit to add a couple of changes that were worthwhile for me :

- A new parameter -skippatches that skips downloading the patches (patches meaning files starting with patch% for the time being)
Ideally we would like to download the installer and any patch more up-to-date that that.

- A new parameter -onelanguage that downloads the files just for one language in -lang order. In my -lang es en case meaning that will only download the spanish installer. If there isn't the it would download the english installer.

(both very straighforward additions, as I don't know python at all).

Instead of:
downloadsOS = [game_item for game_item in item.galaxyDownloads if game_item.os_type in os_list]
I changed it to:
downloadsOS = []
for osname in os_list:
for game_item in item.downloads:
if game_item.os_type==osname:
if (not skippatches) or (skippatches and not game_item.name.startswith('patch')):
downloadsOS.append(game_item)
(for all three downloads, galaxyDownloads and sharedDownloads)

And instead of:
downloadslangs = [game_item for game_item in item.downloads if game_item.lang in valid_langs]
I changed it to:
downloadslangs = []
for validlang in valid_langs:
for game_item in item.downloads:
if game_item.lang==validlang:
downloadslangs.append(game_item)
if (onelanguage) and (downloadslangs):
break
(also for all three downloads, galaxyDownloads and sharedDownloads)
I think the forum ate your spacing which is super bad for python code.

Unfortunately this forum doesn't support code tags so you may want to try pastebin

If you want to make a pull request on the dev branch I'll take them too (though I might edit it a bit). I've been meaning to add something like this. Th
Post edited August 12, 2018 by Kalanyr
Posted August 12, 2018
Some days (or a week ago) I was using gogrepo normally and then for what ever reason I started getting something like:


dlf@dlf:/media/dlf/Storage/gogrepo$ ./gogrepo.py login
Username: dbzlotrfan
Password:
13:12:18 | attempting gog login as 'dbzlotrfan' ...
13:12:20 | fatal...
Traceback (most recent call last):
File "./gogrepo.py", line 1167, in <module>
main(process_argv(sys.argv))
File "./gogrepo.py", line 1139, in main
cmd_login(args.username, args.password)
File "./gogrepo.py", line 550, in cmd_login
with request(login_data['auth_url'], delay=0) as page:
File "./gogrepo.py", line 170, in request
page = opener.open(req)
File "/usr/lib/python2.7/urllib2.py", line 421, in open
protocol = req.get_type()
File "/usr/lib/python2.7/urllib2.py", line 283, in get_type
raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: undefined


Why? And how can I fix it? I can't login, I can't download . . . .
Posted August 12, 2018
avatar
dbzlotrfan: Some days (or a week ago) I was using gogrepo normally and then for what ever reason I started getting something like:
You can read this post:
https://www.gog.com/forum/general/gogrepopy_python_script_for_regularly_backing_up_your_purchased_gog_collection_for_full_offline_e/post1634

Or get the last version of the script:
https://www.gog.com/forum/general/gogrepopy_python_script_for_regularly_backing_up_your_purchased_gog_collection_for_full_offline_e/post1641
Posted August 14, 2018
Hi there, short question:

I switched from the original eddie3's version to Kalanyr's branch. I managed to get the missing modules via pip.

I was hoping I could keep the old manifest file? Or do I have to re-import everything? (If the latter, can I simply import the existing directories somehow without copying 2TB around?)

At least now the update command gives me the following:
09:38:28 | ( 1 / 778) fetching game details for 1954_alcatraz...
09:38:30 | ( 2 / 778) fetching game details for a_kiss_for_the_petals_remembering_how_we_met...
09:38:37 | ( 3 / 778) fetching game details for abzu...
09:38:38 | ( 4 / 778) fetching game details for ad_2044...
09:38:40 | error
Traceback (most recent call last):
File "F:\games\__repo\gogrepo.py", line 1254, in cmd_update
handle_game_updates(gamesdb[item_idx], item,strict)
File "F:\games\__repo\gogrepo.py", line 544, in handle_game_updates
for oldDownload in olditem.downloads+olditem.galaxyDownloads+olditem.sharedDownloads:
File "F:\games\__repo\gogrepo.py", line 264, in __getattr__
raise AttributeError(key)
AttributeError: galaxyDownloads
09:38:40 | ( 5 / 778) fetching game details for age_of_wonders...
09:38:41 | -> gog flagged this game as updated
09:38:41 | -> changelog was updated
09:38:41 | error
Traceback (most recent call last):
File "F:\games\__repo\gogrepo.py", line 1254, in cmd_update
handle_game_updates(gamesdb[item_idx], item,strict)
File "F:\games\__repo\gogrepo.py", line 544, in handle_game_updates
for oldDownload in olditem.downloads+olditem.galaxyDownloads+olditem.sharedDownloads:
File "F:\games\__repo\gogrepo.py", line 264, in __getattr__
raise AttributeError(key)
AttributeError: galaxyDownloads
09:38:41 | ( 6 / 778) fetching game details for ai_war_fleet_command...
09:38:46 | error
Traceback (most recent call last):
File "F:\games\__repo\gogrepo.py", line 1254, in cmd_update
handle_game_updates(gamesdb[item_idx], item,strict)
File "F:\games\__repo\gogrepo.py", line 544, in handle_game_updates
for oldDownload in olditem.downloads+olditem.galaxyDownloads+olditem.sharedDownloads:
File "F:\games\__repo\gogrepo.py", line 264, in __getattr__
raise AttributeError(key)
AttributeError: galaxyDownloads
09:38:46 | ( 7 / 778) fetching game details for akalabeth_world_of_doom...
... goes on and on after that...

Any clue?
Posted August 14, 2018
avatar
toxicTom: I was hoping I could keep the old manifest file? Or do I have to re-import everything? (If the latter, can I simply import the existing directories somehow without copying 2TB around?)
Not sure what you mean by importing, but you can keep your existing downloaded installers even if you decide to delete your old manifest file and get a new one with gogrepo.py update.
Posted August 14, 2018
avatar
timppu: Not sure what you mean by importing, but you can keep your existing downloaded installers even if you decide to delete your old manifest file and get a new one with gogrepo.py update.
Ta, managed to get it to run... Has been a while - ~ 1TB of new games and updates 0_o
Posted August 14, 2018
avatar
toxicTom: Ta, managed to get it to run... Has been a while - ~ 1TB of new games and updates 0_o
LOL. so yeah it has apparently been awhile since you last ran gogrepo update? :)

On the bright side, you have probably also skipped masses of former updates that are now obsolete.
Posted August 14, 2018
avatar
timppu: LOL. so yeah it has apparently been awhile since you last ran gogrepo update? :)

On the bright side, you have probably also skipped masses of former updates that are now obsolete.
More than a year... and I bought a ton of stuff...
Posted August 15, 2018
avatar
toxicTom: Hi there, short question:

I switched from the original eddie3's version to Kalanyr's branch. I managed to get the missing modules via pip.

I was hoping I could keep the old manifest file? Or do I have to re-import everything? (If the latter, can I simply import the existing directories somehow without copying 2TB around?)

At least now the update command gives me the following:

09:38:28 | ( 1 / 778) fetching game details for 1954_alcatraz...
09:38:30 | ( 2 / 778) fetching game details for a_kiss_for_the_petals_remembering_how_we_met...
09:38:37 | ( 3 / 778) fetching game details for abzu...
09:38:38 | ( 4 / 778) fetching game details for ad_2044...
09:38:40 | error
Traceback (most recent call last):
File "F:\games\__repo\gogrepo.py", line 1254, in cmd_update
handle_game_updates(gamesdb[item_idx], item,strict)
File "F:\games\__repo\gogrepo.py", line 544, in handle_game_updates
for oldDownload in olditem.downloads+olditem.galaxyDownloads+olditem.sharedDownloads:
File "F:\games\__repo\gogrepo.py", line 264, in __getattr__
raise AttributeError(key)
AttributeError: galaxyDownloads
09:38:40 | ( 5 / 778) fetching game details for age_of_wonders...
09:38:41 | -> gog flagged this game as updated
09:38:41 | -> changelog was updated
09:38:41 | error
Traceback (most recent call last):
File "F:\games\__repo\gogrepo.py", line 1254, in cmd_update
handle_game_updates(gamesdb[item_idx], item,strict)
File "F:\games\__repo\gogrepo.py", line 544, in handle_game_updates
for oldDownload in olditem.downloads+olditem.galaxyDownloads+olditem.sharedDownloads:
File "F:\games\__repo\gogrepo.py", line 264, in __getattr__
raise AttributeError(key)
AttributeError: galaxyDownloads
09:38:41 | ( 6 / 778) fetching game details for ai_war_fleet_command...
09:38:46 | error
Traceback (most recent call last):
File "F:\games\__repo\gogrepo.py", line 1254, in cmd_update
handle_game_updates(gamesdb[item_idx], item,strict)
File "F:\games\__repo\gogrepo.py", line 544, in handle_game_updates
for oldDownload in olditem.downloads+olditem.galaxyDownloads+olditem.sharedDownloads:
File "F:\games\__repo\gogrepo.py", line 264, in __getattr__
raise AttributeError(key)
AttributeError: galaxyDownloads
09:38:46 | ( 7 / 778) fetching game details for akalabeth_world_of_doom...
avatar
toxicTom: ... goes on and on after that...

Any clue?
You are supposed to be able to use an old manifest and it will update as necessary on the fly. If that isn't happening it's a bug. I suspect I forgot to account for a partial manifest when I added a feature given when it happened. I'll check and fix it.

Edit - Yup. I forgot that checking the historical data of an item may not be converted even if the current data is. Fixed it locally, so it'll roll to dev next time I update, not sure if it's worth backporting,
Post edited August 15, 2018 by Kalanyr
Pages:
1
50
100
108
109
110
111
112
200
250
284
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4246 posts)(4246 posts)
(4246 posts)