It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3861 posts)(3861 posts)
(3861 posts)
Pages:
1
50
95
96
97
98
99
150
200
250
258
This is my favourite topic
Posted March 17, 2018
avatar
Kalanyr: I'm going to do my best to find a proper solution first.
avatar
timppu: Sorry I missed this, but what problem are you trying to solve here now? That the system doesn't automatically go into sleep or hibernate mode, if gogrepo is running?

I didn't even realize there is such an issue, I've been happily using gogrepo both in Windows 7 and Linux Mint without such issues. Is this some Windows 10 thing, it wants to go to sleep whenever it can?

EDIT: Then again, I guess I have changed the Windows 7 power settings so that it doesn't go to sleep mode on its own, and I have removed the hibernate mode altogether. I hate those things, I never want them to kick in on their own.
Stopping the system suspending/hibernating if GOGrepo is running (not for critical battery etc, those are higher priority), there's ways to do it on Windows / Mac that rely on their respective native APIs (which is not great but it's something) , but there's no even semi-standardized method on *nix, it depends on at least one of : a) The desktop manager (gnome / kde /etc) b) systemd (system suspend inhibits which may also require running as root because the system power management daemon treats all of its operations as privileged (which is generally sensible but probably not for something that can be simulated by jiggling the mouse) and they all rely on dbus which doesn't have a currently maintained full python implementation for bonus points).
Posted March 17, 2018
For Mac users:

I've just pushed a new version of gogrepoc to the dev branch. If you could enable your Mac's suspend/sleep powersave (while plugged in, battery has higher priority) and check that it a) prevents sleep while running, b) allows it to go to sleep (after the standard period has passed) when done and c) works on both python 2.7/3 I'd appreciate it.

Please do check the last few commits first before doing so, some of the command syntax and behaviour is different.

If you're using a brew python install you may have to install pyobjc (it should be installed by default on the python that Apple installs as it's necessary for the Cocoa GUI).
Post edited March 17, 2018 by Kalanyr
Posted March 19, 2018
So for those who use Linux on Dekstop/Laptop

I think the best way to handle power management on these systems at the moment is PyQt5s Dbus implementation (it's complete and supported and not distro specific which is more than I can say for literally any other Python implementation at the moment).

Is it a reasonable assumption that everyone has Qt5 installed on their systems* ? I would think so since it's used in a lot of Linux programming and Qt5 has been out for over 5 years now.

*I realize it may not be safe for remote servers , but I suspect power management isn't necessary on those either, so this is more a personal desktop/laptop kind of question.
Posted March 19, 2018
For Linux
I've decided to use PyQt5 and to only use the logind suspend Inhibit to start with which should work on all major distros hopefully.

If you use a distro using Gnome/MATE/KDE or you use a no systemd distro and find this doesn't work for you once I've released it, let me know and I'll see about support for whatever weird ass thing they do on DBus. I'm using OpenSUSE atm so I know it supports the logind method despite also using the power manager of its Desktop (which is KDE in my case). I'm hoping that sanity has prevailed and that's generally the case, it seems that a standard user has appropriate permissions for this in a normal desktop environment too which is good

PyQt5 isn't without its flaws though, it's impossible to manually release the system wakelock though it will be released when python exits , which is undesirable but acceptable. And the way to validate you've got a correct interface that supports certain methods is hilariously awful as far as I can tell. It really should be handled under the hood like dbus-python and pydbus do but since the former is unmaintained and the latter can't handle file descriptors it's the best I've got for now.
I'll keep an eye on pydbus and likely move to it when it's more complete.
It's amazing how ridiculously bad programmatic power management is on Linux, I suppose it's a consequence of being used on servers more than desktops and the desktop user space being fragmented between distros. I suppose I should be greatly there's a generic method at all , at the moment.
Post edited March 19, 2018 by Kalanyr
Posted March 20, 2018
Okay. New Wakelock stuff pushed to Dev.

Linux Wakelock is implemented , and requires PyQt5 (this is optional, you just won't get the suspend block without it). PyQt5 is available from pip for Python3 , and an unofficial version for Python2 (Python-Qt5) can also be installed from pip.
Most major distros should also provide these in their package managers and those should be preferred where available (especially for Python 2). This also requires a recent systemd and dbus as well as the freedesktop login manager and introspection dbus services being available, but I believe this should be the case on nearly all modern distros (systemd-less variants excepted and maybe Gnome). If you find that this doesn't prevent suspend please let me know and I'll add support for the appropriate dbus service for your system.

I'd really appreciate some testing of the Linux (non-SUSE) and Mac wakelock versions, as I can't test those. I know that Windows is working and that the Linux wakelock works on OpenSuse.

Next I'll see about the update function and then fixing the documentation.
Posted March 22, 2018
I've been using Kalanyr's version of this script to run every night on my NAS, and the download step seems to run for an hour or two before bailing out. It picks up where it left off the next night, which is great, but after looking through the logs I'm seeing a lot of exceptions occurring throughout the run. I tried searching this thread and googling for references to these errors, but couldn't find anything, so I thought I'd post here and see if this is expected behavior or not.

I'll see a ton of this error in a row:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 280, in recv_into
return self.connection.recv_into(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1335, in recv_into
self._raise_ssl_error(self._ssl, result)
File "/usr/local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1149, in _raise_ssl_error
raise WantReadError()
OpenSSL.SSL.WantReadError
Then errors like this:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 280, in recv_into
return self.connection.recv_into(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1335, in recv_into
self._raise_ssl_error(self._ssl, result)
File "/usr/local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1166, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (54, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 302, in _error_catcher
yield
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 384, in read
data = self._fp.read(amt)
File "/usr/local/lib/python3.6/http/client.py", line 449, in read
n = self.readinto(b)
File "/usr/local/lib/python3.6/http/client.py", line 493, in readinto
n = self.fp.readinto(b)
File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 296, in recv_into
return self.recv_into(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 296, in recv_into
return self.recv_into(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 296, in recv_into
return self.recv_into(*args, **kwargs)
[Previous line repeated 87 more times]
File "/usr/local/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 285, in recv_into
raise SocketError(str(e))
OSError: (54, 'ECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 747, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 436, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 401, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 320, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: OSError("(54, \'ECONNRESET\')",)', OSError("(54, 'ECONNRESET')",))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "gogrepo.py", line 1736, in worker
dlsz = ioloop(tid, path, response, out)
File "gogrepo.py", line 1559, in ioloop
for chunk in response.iter_content(chunk_size=4*1024):
File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 750, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: OSError("(54, \'ECONNRESET\')",)', OSError("(54, 'ECONNRESET')",))
At some point, the latter set of errors is followed by the download process exiting. It seems to ignore these errors for a while, continuing with the download process, but always eventually exits before everything is completed. Any ideas what might be going on here?

Thanks for any and all help =)
Post edited March 22, 2018 by KuroKitten
Posted March 23, 2018
avatar
KuroKitten: I've been using Kalanyr's version of this script to run every night on my NAS, and the download step seems to run for an hour or two before bailing out. It picks up where it left off the next night, which is great, but after looking through the logs I'm seeing a lot of exceptions occurring throughout the run. I tried searching this thread and googling for references to these errors, but couldn't find anything, so I thought I'd post here and see if this is expected behavior or not.

I'll see a ton of this error in a row:

At some point, the latter set of errors is followed by the download process exiting. It seems to ignore these errors for a while, continuing with the download process, but always eventually exits before everything is completed. Any ideas what might be going on here?

Thanks for any and all help =)
It may be a pyopenssl error, I know from experience the library is imperfect.

But it superficially appears your connection to the GOG server is being reset and it's running retries and moving on to the next download until everything has failed.

I'd recommend making sure pyopenssl / requests / python 3 and your systems ssl lib and certificates are all up to date and making sure other long running SSL connections don't get reset.

ETA: I should probably check for errors like this and reinitialize the session when they happen but that would take some refactoring and would involve making the in use session globally changable either by making it a global variable or by indirection.

ETA2. - And the way logging in works.
Post edited March 26, 2018 by Kalanyr
Posted March 27, 2018
Umm.. Gog chat is (still?) broken and gives me the error 404.

Empress_Owl isn't capable to use the command prompt, but maybe I could create a simple batch script for him, so he could just run it without much user interaction..

@Empress_Owl
Is your system Windows? 7 or 10?
Are you capable to install programs (Python) and edit simple scripts (e.g: paths)?
Which languages do you use? Only french?
In what path are you going to save your games?
Are you going to save also the extras?
Post edited March 27, 2018 by phaolo
Posted March 29, 2018
I know I said the next update would be the update command but it turns out I lied. I needed to cleanup my !orphaned directory today, so I've pushed a build with the trash command to the dev branch.

This is a command that cleans up the !orphaned directory, it can be set to delete only -onlyinstallers (exe,bin,dmg,sh)* or to nuke everything in that directory.

*This is so you can double check GOG hasn't removed without replacing certain files like manuals etc (generally these are removed because they are so outdated as to be worthless if they aren't replaced but this way lets you manually check that stuff more easily since you can nuke the installers first then just check through the few remaining directories/files).
Posted April 03, 2018
Kalanyr, thanks a lot for all the work you added on gogrepo.
I am trying to use it: (windows 8.1, c:\python27 ) : it complained it does not have "requests" installed.
I managed to : c:\Python27\python.exe -m pip install requests # it worked.
But now it apparently needs an openssl package... and I apparently don't have any

c:\Python27\python.exe -m pip install openssl (or OpenSSL, or openSSL) complains there are no packages named like that.

c:\Python27\python.exe -m pip search ssl | find /I "openssl" does not show an openssl package (it shows many packages, all depending on openssl)

c:\Python27\python.exe -m pip install crytto # or another openssl dependant package) fails so far

Can someone "hand guide me" into installing properly python, the packages needed, and the basics of that new version of gogrepo? (I can do the latter part probably, once I managed to get everything needed.... but I am new to python, I don't really "grok" yet how it is using & finding libraries).
(and should I stick with 2.x ? the README.md says "v3 compatibility" will come soon, I assume it is not yet fully supported?)


the latest commands:

G:\__games__\GOG\Gog_Downloads\extras>c:\Python27\python.exe gogrepo.py
Traceback (most recent call last):
File "gogrepo.py", line 42, in <module>
import OpenSSL
ImportError: No module named OpenSSL

G:\__games__\GOG\Gog_Downloads\extras>c:\Python27\python.exe -m pip install OpenSSL
Collecting OpenSSL
Could not find a version that satisfies the requirement OpenSSL (from versions: )
No matching distribution found for OpenSSL

G:\__games__\GOG\Gog_Downloads\extras>c:/Python27/python.exe -m pip install pypandoc
Collecting pypandoc
Downloading pypandoc-1.4.tar.gz
Requirement already satisfied: setuptools in c:\python27\lib\site-packages (from pypandoc)
Requirement already satisfied: pip>=8.1.0 in c:\python27\lib\site-packages (from pypandoc)
Collecting wheel>=0.25.0 (from pypandoc)
Downloading wheel-0.31.0-py2.py3-none-any.whl (41kB)
100% |################################| 51kB 358kB/s
Installing collected packages: wheel, pypandoc
Running setup.py install for pypandoc ... done
Successfully installed pypandoc-1.4 wheel-0.31.0

G:\__games__\GOG\Gog_Downloads\extras>c:/Python27/python.exe -m pip install crytto
Collecting crytto
Using cached crytto-0.2.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\users\me\appdata\local\temp\pip-build-q20z5i\crytto\setup.py", line 7, in <module>
long_description = convert_file('README.md', 'rst')
File "c:\Python27\lib\site-packages\pypandoc\__init__.py", line 137, in convert_file
raise RuntimeError("source_file is not a valid path")
RuntimeError: source_file is not a valid path

----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in c:\users\me\appdata\local\temp\pip-build-q20z5i\crytto\


What am I missing? :/
Post edited April 04, 2018 by EdhelDil
Posted April 04, 2018
avatar
EdhelDil: What am I missing? :/
The package you need is called pyOpenSsl
Posted April 04, 2018
avatar
EdhelDil: What am I missing? :/
avatar
Revan67: The package you need is called pyOpenSsl
many thanks, good sir, it worked! (the descriptions seemed like a "wrapper", so I didn't try that one).
Thanks a lot!
Posted April 04, 2018
avatar
Revan67: The package you need is called pyOpenSsl
avatar
EdhelDil: many thanks, good sir, it worked! (the descriptions seemed like a "wrapper", so I didn't try that one).
Thanks a lot!
No problem, I got stuck on that one a while back too
Posted April 05, 2018
I think the readme on github needs to be updated, had to reinstall my os along with python and now I'm having a hell of a time figuring out what modules I need to reinstall with pip to get this to work.

I should write it down at some point :D
Posted April 05, 2018
avatar
mmodri: I think the readme on github needs to be updated, had to reinstall my os along with python and now I'm having a hell of a time figuring out what modules I need to reinstall with pip to get this to work.

I should write it down at some point :D
Yea that would be a good thing
Pages:
1
50
95
96
97
98
99
150
200
250
258
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3861 posts)(3861 posts)
(3861 posts)