It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3896 posts)(3896 posts)
(3896 posts)
Pages:
1
50
65
66
67
68
69
150
200
250
260
This is my favourite topic
Posted November 19, 2017
avatar
Kalanyr: I think I've preserved one for one backwards compatibility* (assuming you use Windows / English / Standalone installers. You may have to make some minor adjustments otherwise).
I've added some extra functionality that might reduce some of the need for your batch files depending what they do though (it's generally more fine grained control of exclusion / inclusion). Have a look at the help.

*The commands may do some extra stuff like logging, but they should carry out their original purpose and side effects should be relatively minor.

I also think this update will fix that outstanding issue where Python3 no longer worked for my fork after the last update (sorry about that Python3 peeps). I'm just testing that at the moment.

Edit - Yes, it does fix the Python3 issue. Not however that Python2 manifest files probably are not compatible with python3 and may need to be regenerated (chances are excellent that a Python2 manifest contains a size that was a Long in python2 (encoded as size: xxxxxxL) , python3 doesn't have longs (since the long and int types have been unified). Unfortunately Python3 can't recognize a Python2 Long stored in manifest and treat it as an Int. Python2 on the other hand transparently turns any Long sized Int into a Long automatically). So basically don't use Python2 / Python3 interchangeably on the same manifest if you want to avoid grief.
Also, the readme on yours looks identical to the readme on the original. How do I know what command flags change and their uses?
Posted November 19, 2017
avatar
Kalanyr: I think I've preserved one for one backwards compatibility* (assuming you use Windows / English / Standalone installers. You may have to make some minor adjustments otherwise).
I've added some extra functionality that might reduce some of the need for your batch files depending what they do though (it's generally more fine grained control of exclusion / inclusion). Have a look at the help.

*The commands may do some extra stuff like logging, but they should carry out their original purpose and side effects should be relatively minor.

I also think this update will fix that outstanding issue where Python3 no longer worked for my fork after the last update (sorry about that Python3 peeps). I'm just testing that at the moment.

Edit - Yes, it does fix the Python3 issue. Not however that Python2 manifest files probably are not compatible with python3 and may need to be regenerated (chances are excellent that a Python2 manifest contains a size that was a Long in python2 (encoded as size: xxxxxxL) , python3 doesn't have longs (since the long and int types have been unified). Unfortunately Python3 can't recognize a Python2 Long stored in manifest and treat it as an Int. Python2 on the other hand transparently turns any Long sized Int into a Long automatically). So basically don't use Python2 / Python3 interchangeably on the same manifest if you want to avoid grief.
avatar
paladin181: I create my own logs, are your logs stored in the parent folder of the gogrepo script?
Yes, as gogrepo.log , and gogrepo,.log1-9 , each limited to a maximum of 10 MB in size (which is more than enough to do several full manifest build / download / verify with my pretty massive collection).

You can use the -h flag , both as gogrepo -h , to see all the high level commands and gogrepo command -h to get detailed info about the command .
Post edited November 19, 2017 by Kalanyr
Posted November 19, 2017
avatar
Kalanyr: Yes, as gogrepo.log , and gogrepo,.log1-9 , each limited to a maximum of 10 MB in size (which is more than enough to do several full manifest build / download / verify with my pretty massive collection).

You can use the -h flag , both as gogrepo -h , to see all the high level commands and gogrepo -h command to get info about command .
Thanks for getting back.
Posted November 19, 2017
avatar
Pitney_b: thanks for the quick response. i'll revert to my old script for the moment so there is no rush.
Okay, all those issues should be fixed. Let me know if you find any further problems.
Posted November 20, 2017
avatar
Pitney_b: thanks for the quick response. i'll revert to my old script for the moment so there is no rush.
avatar
Kalanyr: Okay, all those issues should be fixed. Let me know if you find any further problems.
Thanks Kalanyr everything appears to be working ok.
Posted November 20, 2017
Every couple of weeks I build a new manifest and download dryrun output file. This weekend it said there were 74 games to update. After manually comparing the file names I found all but 8 of them were false positives.
Posted November 20, 2017
Just as a heads up I've merged working2 back into the master branch on my fork. I was keeping that to track compatability with woolymethods repo but I've decided that it's not worth it at that stage, if/when he comes back with the python3 update it'll likely be so different in structure as to be irreconcilable in any case.

avatar
SKARDAVNELNATE: Every couple of weeks I build a new manifest and download dryrun output file. This weekend it said there were 74 games to update. After manually comparing the file names I found all but 8 of them were false positives.
A game is marked updated when any change is made to the executable files / patches or probably the Changelog (but not extras / misc files). This happens even if the update is for a OS/Language you don't use (because the update tag shows before GOG can determine the Language/OS you use for that game). That's likely the issue.
Post edited November 20, 2017 by Kalanyr
Posted November 20, 2017
avatar
Kalanyr: A game is marked updated when...
Sorry if I wasn't clear. I have an update.bat to run for specific games when I see them flagged. That's not what I meant.

I semi-regularly delete the manifest file and build a new one. This gets the latest information when the flag doesn't appear to have been set. I also filter for "-os windows -lang en". No problems there.

After that I run "download -dryrun -skipextras > Downloads.txt" to get a report. Now that I've scrolled back through this thread I think the symbol characters changing caused the file names to not match up, leading it to report there was more to download than necessary.

I've downloaded the working2 version from your link 16 post back. I actually have 2 archives. One is structured and the other managed by GOGRepo. At the moment I'm copying all the EXE and BIN files from the structured one to a single location so your version can tear through it and rebuild the managed one.
Posted November 20, 2017
avatar
Kalanyr: Okay my fork has been updated here:

https://github.com/Kalanyr/gogrepo/tree/working2

Apologies for the delay again. Had some issues with my GOG HDD. And then spend some time trying to work out if I could fix the Python2/3 issue with encoded Longs programatically but I couldn't find anything useful on this issue.
Your version creates a log file which I like. Buy I do wish gog made a proper tool to do this though. I hate using python for this kind of job.
Posted November 20, 2017
avatar
Magmarock: Your version creates a log file which I like. Buy I do wish gog made a proper tool to do this though. I hate using python for this kind of job.
I love using python for it because it gives me more options in which OSes or devices to run it. As I've said before, I've been thinking of setting up a dedicated low-power gogrepo machine (whose only purpose was to keep my GOG collection up to date), for instance with a Rasperry Pi or some cheap Android tablet. Not sure if gogrepo could be run on Android, I haven't really checked that yet, but at least better than some dedicated Windows application.

EDIT: At least there seems to be python for Android, so that's a start... Next obstacle would be is there a way to use external USB NTFS (or some other widely-used file system) hard drives on Android tables, similarly like you can connect USB flash drives to them.
Post edited November 20, 2017 by timppu
Posted November 20, 2017
avatar
Magmarock: Your version creates a log file which I like. Buy I do wish gog made a proper tool to do this though. I hate using python for this kind of job.
avatar
timppu: I love using python for it because it gives me more options in which OSes or devices to run it. As I've said before, I've been thinking of setting up a dedicated low-power gogrepo machine (whose only purpose was to keep my GOG collection up to date), for instance with a Rasperry Pi or some cheap Android tablet. Not sure if gogrepo could be run on Android, I haven't really checked that yet, but at least better than some dedicated Windows application.

EDIT: At least there seems to be python for Android, so that's a start... Next obstacle would be is there a way to use external USB NTFS (or some other widely-used file system) hard drives on Android tables, similarly like you can connect USB flash drives to them.
Yeah, there's an Android app by Chainfire that lets you mount USB drives from your phone/tables USB power/data port. Though it only works on some models since not all phones have the hardware for their USB connection to run in master mode instead of slave. You'd want to test it before using it for anything large though, USB2.0 speeds would be prohibitively slow for a collection the size of yours.
Posted November 20, 2017
avatar
Kalanyr: Yeah, there's an Android app by Chainfire that lets you mount USB drives from your phone/tables USB power/data port. Though it only works on some models since not all phones have the hardware for their USB connection to run in master mode instead of slave. You'd want to test it before using it for anything large though, USB2.0 speeds would be prohibitively slow for a collection the size of yours.
Yeah even my ancient ASUS TF101 Transformer tablet has a full-size USB port on its keyboard dock-station, and it reads at least USB flash drives. I don't remember if it reads only FAT, or also NTFS. I don't recall if I ever tried USB hard drives with it.

I was originally thinking of turning it into a dedicated gogrepo machine, maybe I'll still look into it. It has been out of use since Google released an update the pretty much bricked many older Android devices (including TF101), but IIRC that has been fixed, I haven't just gotten around to apply the update because it has to be done manually.

EDIT: And USB 2.0 speeds are ok to me, I am using USB 2.0 on my PC as well because USB 3.0 has some serious issues (I don't know if it is specifically a Windows 7 issue). If I try to move or copy lots of data over USB 3.0, eventually the copy operation fails as Windows suddenly disconnects the USB hard drives etc. etc. When using USB 2.0 (e.g. through a USB 2.0 hub), no problems. It is as if the USB port becomes overwhelmed by the amount of data when using USB 3.0 speeds.

I think USB 2.0 is speedy enough for downloading anyway, USB 3.0 speeds would be meaningful if I am copying stuff locally from device to device. IIRC the theoretical maximum of USB 2.0 is 480 Mbit/s, and I can download at max 50 Mbit/s or so, usually slower.
Post edited November 20, 2017 by timppu
Posted November 20, 2017
avatar
Magmarock: Your version creates a log file which I like. Buy I do wish gog made a proper tool to do this though. I hate using python for this kind of job.
avatar
timppu: I love using python for it because it gives me more options in which OSes or devices to run it. As I've said before, I've been thinking of setting up a dedicated low-power gogrepo machine (whose only purpose was to keep my GOG collection up to date), for instance with a Rasperry Pi
That's all well and good but it's not being updated properly and there's a few forks but I think this could use a little polish is all. An official tool would be nice though I think it should still be small and tiny.
Posted November 20, 2017
avatar
timppu: I am using USB 2.0 on my PC as well because USB 3.0 has some serious issues (I don't know if it is specifically a Windows 7 issue). If I try to move or copy lots of data over USB 3.0, eventually the copy operation fails as Windows suddenly disconnects the USB hard drives etc.
[..] the theoretical maximum of USB 2.0 is 480 Mbit/s, and I can download at max 50 Mbit/s or so, usually slower.
You must have some specific driver\hw problems.
I have a bunch of USB 3.0 on my PC and Win7 behaves normally with them.

Btw, I use Teracopy to automatically check all MD5 when transfering files.
Also, it tells you your current speed.
USB 2.0 basically never reaches the maximum 480 Mbit/s (60MB/s), and the usual average is way worse like 20 MB/s.
Post edited November 20, 2017 by phaolo
Posted November 20, 2017
avatar
phaolo: You must have some specific driver\hw problem.
I have a bunch of Usb 3.0 on my PC and Win7 behaves normally with them.
Could be. I have the most up to date drivers (updated with the SlimDrivers utility) but the problem still persists. I recall earlier googling for this problem, and some suggested some widely used USB 3.0 ports/chips have (had) a design flaw that would cause this, but I am unsure.

Also, the problem does not occur in "normal use". It happens usually when copying lots of data around, like copying 500 gigabytes (or more) of data from one USB 3.0 hard drive to another. At first everything seems to go fine, and then whooops Windows suddenly "loses" the USB hard drive(s), and reconnects them. With USB 2.0, this never happens.

EDIT: If I recall correctly, someone here suggested it could be related to USB 3.0 requiring more power than USB 2.0. Anyway, it happens also with USB hard drives that don't draw their power from the USB port, but have their own power supply.

I've been meaning to install Windows 10 Pro on this laptop to see if the same problem exists there, as I recall Windows 10 has official USB 3.0 support. Just to see if the same problem happens there too with this ASUS G75VW laptop. Anyway, I'd rather keep this laptop as my Windows 7 gaming PC, and move on to Windows 10 only in some future PC.

avatar
phaolo: USB 2.0 basically never reaches the maximum 480 Mbit/s (60MB/s), and the usual average is way worse like 20-30 MB/s.
Yes, that is why I called it "theoretical maximum". The practical maximum is still well above my download speeds. USB 2.0 might become a bottleneck if I had a gigabit internet connection, but I don't.

Anyway, I've had my share of problems using USB for mass storage in the past as well (on older PCs), so I'm kinda hoping that in the future I could mainly use something more advanced and reliable for external hard drives, e.g. eSATA or Thunderbolt. For now I have to put up with USB.
Post edited November 20, 2017 by timppu
Pages:
1
50
65
66
67
68
69
150
200
250
260
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3896 posts)(3896 posts)
(3896 posts)