It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4274 posts)(4274 posts)
(4274 posts)
Pages:
1
50
84
85
86
87
88
150
200
250
285
This is my favourite topic
Posted December 21, 2017
avatar
RupertMurdock: Transfered my test folder from an external to an internal hdd, killed every superfluous process and made a quick test with commits 350b8b962 (=old) and 27da409d (=new).
As expected the same results: the old one downloads flawlessly the new one slows down.
But I noticed that the cpu load of python.exe is much higher running the new file.
The old one needs about 25% of my capacity at the start and stagnates at 4% during the download.
The new one is constantly higher than 30%.
Is this helpful in any way?

What else could I try out, any ideas, anyone?

If we can't find the culprit, I'll stick with the solution I already mentioned, I will use the new versions for updating and verifying and the old ones for downloading. Not a big deal.
That *is* possible if your CPU is very weak, the 4 threads used for downloading are mainly IO bound, the 4 timer threads added in that update are CPU-bound and are regularly created and destroyed to reset the Timer. It explains why this issue doesn't happen to me either, my CPU is pretty bulky. I'll do some poking and see if I can find a less resource intensive way of creating Timer threads.
Posted December 21, 2017
avatar
RupertMurdock: As expected the same results: the old one downloads flawlessly the new one slows down.
But I noticed that the cpu load of python.exe is much higher running the new file.
The old one needs about 25% of my capacity at the start and stagnates at 4% during the download.
The new one is constantly higher than 30%.
I grabbed Kalanyr's version a few days ago.
So far I've downloaded 100Gb and the speed seems very good here.
I'm using an external USB3.0 drive.
And python.exe is using 16% of the CPU on my pc (i5 6600K).
Posted December 21, 2017
avatar
Kalanyr: That *is* possible if your CPU is very weak, the 4 threads used for downloading are mainly IO bound, the 4 timer threads added in that update are CPU-bound and are regularly created and destroyed to reset the Timer. It explains why this issue doesn't happen to me either, my CPU is pretty bulky. I'll do some poking and see if I can find a less resource intensive way of creating Timer threads.
I have a 2.83 GHz quad core. But my other hardware components are outdated. I get more and faster RAM, a new ssd drive and a new usb card next week.
Posted December 21, 2017
avatar
phaolo: I grabbed Kalanyr's version a few days ago.
So far I've downloaded 100Gb and the speed seems very good here.
I'm using an external USB3.0 drive.
And python.exe is using 16% of the CPU on my pc (i5 6600K).
I mirror this. No problem downloading hundred s of GB . Using an external drive. CPU load is about 19%.
Posted December 21, 2017
Just as a headsup I'm going home for Christmas/New Year and only taking my travel laptop with me, which isn't really setup for development, so it'll probably be ~14 days before I do any serious work on this.



avatar
Kalanyr: *snip*
avatar
RupertMurdock: I have a 2.83 GHz quad core. But my other hardware components are outdated. I get more and faster RAM, a new ssd drive and a new usb card next week.
I have a 4.00 Ghz Quad-Core for comparison.
Post edited December 21, 2017 by Kalanyr
Posted December 21, 2017
avatar
Kalanyr: Just as a headsup I'm going home for Christmas/New Year and only taking my travel laptop with me, which isn't really setup for development, so it'll probably be ~14 days before I do any serious work on this.
You added a lot of improvements already, so just have a good Christmas and a Happy New Year :)
Post edited December 21, 2017 by phaolo
Posted December 21, 2017
avatar
Kalanyr: That *is* possible if your CPU is very weak, the 4 threads used for downloading are mainly IO bound, the 4 timer threads added in that update are CPU-bound and are regularly created and destroyed to reset the Timer. It explains why this issue doesn't happen to me either, my CPU is pretty bulky. I'll do some poking and see if I can find a less resource intensive way of creating Timer threads.
I haven't run your version of gogrepo there, but I used to run the original version on a very old Dell laptop which has some 32-bit 1-core Pentium-M CPU I think (running 32-bit Linux Mint). I was using four download threads there.

At least there I didn't notice any download speed issues with wooly's version.
Post edited December 21, 2017 by timppu
Posted December 21, 2017
avatar
Kalanyr: That *is* possible if your CPU is very weak, the 4 threads used for downloading are mainly IO bound, the 4 timer threads added in that update are CPU-bound and are regularly created and destroyed to reset the Timer. It explains why this issue doesn't happen to me either, my CPU is pretty bulky. I'll do some poking and see if I can find a less resource intensive way of creating Timer threads.
avatar
timppu: I haven't run your version of gogrepo there, but I used to run the original version on a very old Dell laptop which has some 32-bit 1-core Pentium-M CPU I think (running 32-bit Linux Mint). I was using four download threads there.

At least there I didn't notice any download speed issues with woolie's version.
You wouldn't , the issue isn't even present with most versions of mine, it appears to be related to the Timer threads I introduced very recently to stop the script form hanging.
Posted December 21, 2017
avatar
Kalanyr: You wouldn't , the issue isn't even present with most versions of mine, it appears to be related to the Timer threads I introduced very recently to stop the script form hanging.
Ah ok, I didn't read all the messages thoroughly.
Posted December 27, 2017
Kalanyr
in your fork, after I download a game, a new !info.txt file is in the newly created game directory, but the installers are still in the !downloading directory.

gogrepo.py
commit/36b845c9834a596f733e13712112fb1a25cb3e07

EDIT:
hmm, odd behavior on first gogrepo.py run.
after manually deleting those files, second run, files moved correctly.
Post edited December 27, 2017 by solar_dome
Posted December 28, 2017
Is there somewhere a binary standalone build for windows available (CLI, not graphical)?

I don't need it myself, but for a friend who has absolutely no linux knowledge.
Posted December 29, 2017
avatar
solar_dome: Kalanyr
in your fork, after I download a game, a new !info.txt file is in the newly created game directory, but the installers are still in the !downloading directory.

gogrepo.py
commit/36b845c9834a596f733e13712112fb1a25cb3e07

EDIT:
hmm, odd behavior on first gogrepo.py run.
after manually deleting those files, second run, files moved correctly.
Unmoved files in !downloading are almost certainly incomplete as they are preallocoted and then data is downloaded and written in, they won't be moved if there's a problem in that process (as they'd be useless). They shouldn't need deleting though, the resume should handle it, unless the file name was invalidated by an update before the resume.
Posted December 30, 2017
I have several games that give me errors when I do an update.

Gwent - this is most likely because there is no standalone installer for it but I'm including it for completeness

request failed: 404 Client Error: Not Found for url: https://cdn-hw.gog.com/secure/gwent/setup_gwent_1.2.10.31_en.exe.xml?ttl=1514731341&hw_l=42&hw_token=1487c94e4f74d3ac270c98e7d468cb40&_token=eyJhbGciOiJBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0.eqVBlRp3_mfIf_NtvP4tXs9_kOS4Bdqf3JBb2lPIKIw5P23QOBV6nA.Ljb-w3HX-FWtVgxXMVzPAQ.s5nHP7Z-QiVXPZNmswV2dMq_Vcr5xfcvrdw4xtGavILy_juCswcfTZRT-28RZCsIu3kTWArpTH2mDMKvPWwDd5fsvDkCgDF2EO5-7DlSMSWTQWyeLbRaurgCNDvtnYHVTI8q4d1i2O3HVyR50lSRC9MpDHO5e4zuALpR9RWi_JDyrAnf_RQObIp6eHBl1SrkGNQ1l6rgi2h0Yj7BTUOJt8YXkfY-SW70gZDeQDfJ2s1h-sfeTbrxA7gVBbe7sqEQiLkmoqrwdx9cfztgqYBPtfn8VGtB_4nP5ZI1w05poas1uBJCkwHMpuGweCrQ8D-9_TFIA7Zah3x0O-Qjjf6ZmHIwrjLBe5-IOPRaPvtUZrjXaKTPs7dzQ6u3CP1LqASZHJwn76r9j2bpWzSnkFDrH9qjlURSlmK7O2x-yf8P2yiQ-wic2nemW-sIYInrIVMFRB-PVa2twtqoWv6VCV3bIg.MgXgRzNZgtgI0cC_wnhZ5Q&source=hw. will not retry.

Sublevel Zero

request failed: 404 Client Error: Not Found for url: https://cdn-hw.gog.com/secure/offline/1443519435/1443519435/49900730541816738/6791/setup_sublevelzero_1.2_gog1_%289875%29.exe?ttl=1514733181&hw_l=103&hw_token=5a63d2e199f3cb2cdfd5a8ab93cbf4a6&_token=eyJhbGciOiJBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0.jugRZGD0Rk8aXhsVu03uDqoAEWwsm3XmomAZ5iyywVpHhreQxgVpEA.K43njTdPQQsktUsKIMaNWA.M2dDLWQJj82X2Kg2w2aX2zoTDOcVKSxUdLIfBeH_pUvbMQsCOkIyGo7BQX11IRAylQOO-_fdrT1ONRmA6cCqrBLpjoCFuL-8oSjad1qKymt0-dK_FINoYUqjpjg0IdwH1xtQYTfy76pJBwTTa3Vri80LaJYmw03b5kTj58uIwd_b4XhWSvU4XWFh4NXYoTRpRAbMAon_UyPoLi1QM8uuMcjmu_v6GzblqcZiyUSDJvI3YuCdRhA0pRQ2Tnd9_Jjbei2Tsfs0lB4e4BYUUUaeuQ_YvCg2_-083h3mlnv0LQ5ZtkeAgUEa2OIZiFf2NNMUK-bY8WHIQN81LAnBvAoyO6fvbjOxp6V9xeMD5XzImjz_U2lTCfLqWdIbjqMx0fNtAkknfOZkQ2bbPCNOdvb8QfBT6YL8T8xn2s1iBFYGnCe2Ge6ya21_0nqqTArsuxoMApv0ydOb-vdx6S9ghF_N39gvblgp1dsPPhG_m7yp99JGwTY9T06IT0NgJ1ZR5s5N5C7BwPu9QOQ6Uw5_UDDQby2AjKU5C7JptaJL1v73iWfocEawZ0z2VYOK52oAumiS.pjJZJNDC9QOaKSi-bTx6HA&source=hw. will not retry.

Wasteland 2 Director's Cut

request failed: 403 Client Error: Forbidden for url: https://cdn-hw.gog.com/secure?ttl=1514733915&hw_l=7&hw_token=adf528cc5763791c49532204c40a7d2c&_token=eyJhbGciOiJBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0.HkvoBy6IT0v2JC3IkbZe7_xtCj53U4YftvVSZNZnf_SSI5RqvQHxUA.ZfE2Pyv-HsUEDgmr4c0jWQ.WC-LZvPzm-0oQgMInxkLTJ67lwGSsY1nkEQVwh2MvyNg0WBNI01HV36sVhGKgnregBbWgARzvYbps8GGmbDmqhE1FpfngeYSmOtC63tAEWXZItlWVuMhmgtcTodctnbfrScz1tXxqogu98IdKFyB0NWvCuGixS0rewm-3DbCHMkm51bjcdQHneUUtdvr5Y3W-E7wGxbbBu7xI9b9F-g3q_B095r-S2VioUHAHsje3o_Ae2oo3m-p-w1TPDpnEP9asMSvSRi4zwQF_mWc3ewVEW48sW3jgDlgeGDM3esIJOkCAJtif9RWyTVxQVrs0Xl0gdKTCs34EaSxCsNa3-AaQgXtEX0SuBnW8laHxLkNj8DBBfEAytG4jxE6fO6nq9ppiQZthN5DWDNNVH8a-EMbvw.tLqsWWGeAr7OOlVlS0_xXw&source=hw. will not retry.

Wasteland 2 Kickstarter

request failed: 403 Client Error: Forbidden for url: https://cdn-hw.gog.com/secure?ttl=1514733926&hw_l=7&hw_token=bec2c80c938b2a10def00fcdeb18dcc8&_token=eyJhbGciOiJBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0.W944y-uOgjxiBjU1C0emiYcy2pyjsF8KLl99nSWKhs9OTsFBSdMmFA.BbWpifUBAmWJNROdTOsgwg.4AbqY25UatCkP6Y-tsTbFcbhNKNb-Xw4Ov9c29SkKyDRDg9bc1kd3wzNPoRGqkharkl8rciQtRPcH0KUFGHmMfZW_FfCkVju23xAOCuTNf38c7fcz1zXBsI1vMPXP18i7MsovWFtmEIxqFFHxrGNLDz0vm2vYdjCUAKQhDpOOi3JOMr0K9PaXSWVxGrcN8RkiiV4NXf-DDliYEbrAeAmVCRNQJ2SkJEMCi0V4i7QbxnfQYj10Ds8dFOkA2-XedgkFKsqT_rnEMvg_Dr-BBeIYRerCQ2yb3o-3RtmL46SMU3Zr4JFsQiQqkaDpYDdVmsoG8NWKefZiZQd7CqdxZ4udjKGSbe_j0eo5gHfXLzvxN0qfzT1qK40QZJCMDsWiorQ1VBT8S2GU1BgukueiwAHZg.j4tV-DGu_l47YnbwhJp1aA&source=hw. will not retry.
Posted December 30, 2017
avatar
KunoNoOni: I have several games that give me errors when I do an update.
Your errors seem similar to these:
https://www.gog.com/forum/general/gogrepopy_python_script_for_regularly_backing_up_your_purchased_gog_collection_for_full_offline_e/post1270

And Kalanyr about the error 404 said:
"The kyn / dreamfall errors are normal. It's missing md5s or files on GOG side."

Not sure about the 403, tho.
Post edited December 30, 2017 by phaolo
Posted January 02, 2018
If you get 404/403 errors check the downloads on GOG.com for that game and see if here's a file which produces the same error.

I'll see about adding a command to list the files in the manifest for a specific game to make cross-check easier. The easiest way currently is probably to download with -ids and -dryrun.
Pages:
1
50
84
85
86
87
88
150
200
250
285
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4274 posts)(4274 posts)
(4274 posts)