It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3859 posts)(3859 posts)
(3859 posts)
Pages:
1
50
100
150
200
211
212
213
214
215
258
This is my favourite topic
Posted June 11, 2022
avatar
Magnitus: I believe most git services have accompanying functionality to manage release binaries which fills in for a lot of use-cases where you want to produce binaries from a codebase (in a lot of my github repos, I just have a release pipeline that generates a release with accompanying binaries from the codebase with the version of the release being based on the git tag).
I meant storing assets like images, textures, sound files, etc.
Storing binaries generated from the versioned source code together with that same code in Git is just ... evil.
I've seen it done, but never got the point.
Posted June 11, 2022
avatar
brouer: I meant storing assets like images, textures, sound files, etc.
Fair enough. Though even then:
- If the binaries file are small and don't change all that much, you can hit a small performance penality and store then in git anyways
- Otherwise, unless you REALLY want to optimize on your storage costs (there are some cases for that though I don't think it is the majority), you can combine your git repo with some generic file storage service (separating versions by path and possibly referencing the versioned path in the code even for downloads purposes).

avatar
brouer: Storing binaries generated from the versioned source code together with that same code in Git is just ... evil.
I've seen it done, but never got the point.
To be clear, in the case of the release service provided by git providers like github, the releases don't live IN the git history. It is simply and adjacent service that lives in the same repo, like this: https://github.com/coredns/coredns/releases

Basically, you trigger a release from your code pipeline, but the release is otherwise uncoupled from your git history (and as long as you follow the sane practice of not changing your git history or overwriting pre-existing release numbers, will faithfully represent your release at that point in time in the code without future remediation on your part). It simply lives in the same repository (repository refering to a meta-structure provided by your git provider comprising your git repository and other adjacent things) for added convenience.

But yes, I would not advocate storing releases IN the git history. That is, as you say, evil.
Post edited June 11, 2022 by Magnitus
Posted June 16, 2022
Is there a way to ignore errors and download all files attributed to a title in gogrepoc? This would be a way to get the current files on those that constantly give us issues on downloading.
Posted June 16, 2022
avatar
Starkrun: Is there a way to ignore errors and download all files attributed to a title in gogrepoc? This would be a way to get the current files on those that constantly give us issues on downloading.
What kind of issues do you see on downloading? I download almost daily and don't see any there, only on update.

Perhaps post the command line you use and a few error messages you get.
Post edited June 16, 2022 by mrkgnao
Posted June 17, 2022
avatar
Starkrun: Is there a way to ignore errors and download all files attributed to a title in gogrepoc? This would be a way to get the current files on those that constantly give us issues on downloading.
There shouldn't be any blocks due to errors where the files are actually available for download and non-corrupt. If an error is actually blocking the download it's because the error is fatal.
Posted June 20, 2022
avatar
Starkrun: Is there a way to ignore errors and download all files attributed to a title in gogrepoc? This would be a way to get the current files on those that constantly give us issues on downloading.
avatar
Kalanyr: There shouldn't be any blocks due to errors where the files are actually available for download and non-corrupt. If an error is actually blocking the download it's because the error is fatal.
I dug into my logs and saw that. I'm actually TEE'ing (piping) everything to individual logs now so I can find stuff easier. Sorry about that and thanks for replying!
python gogrepoc.py update -skiphidden -updateonly | tee -Append UpdateLog.txt
python gogrepoc.py update -skiphidden -skipknown | tee -Append UpdateSkipLog.txt
python gogrepoc.py clean g:\gog | tee -Append cleanLog.txt
python gogrepoc.py download g:\gog | tee -Append downloadLog.txt
if ((read-host "Would you like to verify all now? Y/N") -like "*Y*"){python gogrepoc.py verify g:\gog | tee postVerify.txt}
Posted June 20, 2022
avatar
mrkgnao: I have added ymgve's "extra slashes" solution (see here) to my copy of gogrepoc. It solved all my current MD5 XML issues.

I didn't try to add it to the GitHub repository because:
(a) it's a hack
(b) when I retired, I swore I would never use git ever again

So, if you want it, you can add it to your own local copy.

To do so:
1) Search for the following line:
tmp_md5_url = append_xml_extension_to_url_path(response.url)
2) Add the following two lines after it (with the same indentation!):
extra_slashes = '/' * (datetime.datetime.today().minute + 2) ### RS: Setting replacement string based on current minute 09/06/22
tmp_md5_url = extra_slashes.join(tmp_md5_url.rsplit('/',1)) ### RS: Adding extra final slashes to bypass GOG's XML issues 09/06/22
3) Save

What it does is add extra slashes based on the current minute (i.e. between 1 and 60 extra slashes), so if you you have an XML failure on a specific game, wait a minute and try just that game again.

I had 6 games with outstanding XML failures (most had been failing for weeks). I ran gogrepoc on these 6 games with the fix and 5 were resolved. I ran it on the 6th game again a minute or two later and it was resolved too.

This handles failures during update. There could also be failures during download, but I have very rarely seen any. If you have those, you can use a similar solution there too. I haven't tried it on real life failures, but I did add it to my code nonetheless and it doesn't seem to hurt anything.

To do that (optional):
1) Search for the following line:
chunk_url = append_xml_extension_to_url_path(response.url)
2) Add the following two lines after it (with the same indentation!):
extra_slashes = '/' * (datetime.datetime.today().minute + 2) ### RS: Setting replacement string based on current minute 09/06/22
chunk_url = extra_slashes.join(chunk_url.rsplit('/',1)) ### RS: Adding extra final slashes to bypass GOG's XML issues 09/06/22
3) Save

Thank you, ymgve.

P.S. I would appreciate a code review, if someone is up to it.
Is there an updated version I should be using? I only have
tmp_md5_url = response.url.replace('?', '.xml?')
Posted June 20, 2022
avatar
DDDespair: Is there an updated version I should be using? I only have
tmp_md5_url = response.url.replace('?', '.xml?')
Yes. You're using an older version. You can get the current version here:
https://github.com/Kalanyr/gogrepoc/tree/master
Post edited June 20, 2022 by mrkgnao
Posted June 26, 2022
avatar
DDDespair: Is there an updated version I should be using? I only have
tmp_md5_url = response.url.replace('?', '.xml?')
avatar
mrkgnao: Yes. You're using an older version. You can get the current version here:
https://github.com/Kalanyr/gogrepoc/tree/master
Is the update still not working? I have the newest build, and added the two lines as you recommned, and I still get this error :(

09:52:55 | ( 5 / 258) fetching game details for age_of_wonders_shadow_magic...
09:52:55 | The handled exception was:
09:52:55 | error
Traceback (most recent call last):
File "D:\Backup\GOG\gogrepoc.py", line 1379, in cmd_update
item_json_data = response.json()
File "C:\Program Files\Python39\lib\site-packages\requests\models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Program Files\Python39\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Program Files\Python39\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\Python39\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
09:52:55 | End exception report.

Am I doing something wrong?
Posted June 26, 2022
avatar
MMLN: Am I doing something wrong?
This issue doesn't seem related to MD5 XML issue. It's something new or temporary.

Try running an update on this game alone (using -ids age_of_wonders_shadow_magic) and see if it repeats.

P.S. I tested the game myself and saw no issues.
Post edited June 26, 2022 by mrkgnao
Posted June 29, 2022
Has Kalanyr updated gogrepo with ymgve's "hack", or should we still manually edit it as explained by mrkgnao?
Posted June 29, 2022
avatar
phaolo: Has Kalanyr updated gogrepo with ymgve's "hack", or should we still manually edit it as explained by mrkgnao?
You can always check the script's latest version on GitHub. Latest is March 21, so it seems my changes are still required.
Posted June 29, 2022
avatar
phaolo: Has Kalanyr updated gogrepo with ymgve's "hack", or should we still manually edit it as explained by mrkgnao?
avatar
mrkgnao: You can always check the script's latest version on GitHub. Latest is March 21, so it seems my changes are still required.
Uh, but I did.
However, there's only one release there:

Testy McTest Latest
This is a test release for work on the built in updater. Please don't use it.

O_o ??

(ah maybe you were referring to the latest file update. Still, it seems a weird method with no releases)
Post edited June 29, 2022 by phaolo
Posted June 29, 2022
Does this actually still work?
All I get is a bunch of "xml parsing error occurred trying to get md5 data" errors for just about every game when trying to generate the manifest.

Also there appear to be several forks of this, such as eddie3 and Kalanyr. Which one should I use?


avatar
Orkhepaj: i wonder who still buys here , each year they make everything more annoying
fe why ruin this downloader...
What's the alternative... DRM'd stores lol?
Post edited June 29, 2022 by SargonAelther
Posted June 29, 2022
avatar
SargonAelther: Does this actually still work?
All I get is a bunch of "xml parsing error occurred trying to get md5 data" errors for just about every game when trying to generate the manifest.

Also there appear to be several forks of this, such as eddie3 and Kalanyr. Which one should I use?
Yes. It works. Use Kalanyr master.

The xml MD5 errors are GOG issues. See my post above for a temporary fix (may require more than one attempt).

https://www.gog.com/forum/general/gogrepopy_python_script_for_regularly_backing_up_your_purchased_gog_collection_for_full_offline_e/post3167

avatar
mrkgnao: You can always check the script's latest version on GitHub. Latest is March 21, so it seems my changes are still required.
avatar
phaolo: Uh, but I did.
However, there's only one release there:

Testy McTest Latest
This is a test release for work on the built in updater. Please don't use it.

O_o ??

(ah maybe you were referring to the latest file update. Still, it seems a weird method with no releases)
I also prefer formal releases and version numbers, but I can live with just looking at the latest file update, given that this is volunteer work on Kalanyr's part.
Post edited June 29, 2022 by mrkgnao
Pages:
1
50
100
150
200
211
212
213
214
215
258
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3859 posts)(3859 posts)
(3859 posts)