It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
We could really use an update function within gogrepo as well.
Okay quick straw poll:

Do people prefer download to grab everything in the manifest or have the same defaults as update ?

This is one of those things I've been thinking about but should settle on before I redo the readme. The current way breaks backward compatability with the original in a way that's strictly unnecessary (it just required more work on my part than I was willing to do at the time) whereas I have generally mantained backwards compatability as much as possible. And I think I've worked out most of the required changes while thinking it over. On the other hand reverting to the old way breaks compatability with scripts people have setup since (though I suspect only for a tiny proportion of the user base who download non-English / non-Windows builds).

I'm leaning towards defaulting to the entire manifest (but keeping the filters available of course).

avatar
kohlrak: We could really use an update function within gogrepo as well.
Hmmm. I'd have to reorganize my repo a little bit and make an explicit test/dev repo as well as a stable one since I tend to use this thread to have some guinea pigs to test stuff I missed but that would be distinclty unhelpful for an internal update. But otherwise it shouldn't be overly difficult.

ETA
To Do List
Fix crash on download where manifest is not empty but where language/os filtered manifest is.
Bandwidth Limiter
Resolve identical installer parts across multiple languages somehow.
Built In Updater for Script.
Post edited March 02, 2018 by Kalanyr
this looks useful but i don't know what python is. some background may be useful
avatar
Kalanyr: The 2nd is GOG side but not really an error as such, it's an item promised as part of Kickstarter that hasn't been provided to GOG (usually such things have ended up on Backerkit or the companies own websites, some of them just haven't happened).
Huh, that's strange. If GOG never had it, how does gogrepo pick up on it? I guess they created a database entry in anticipation or something.
On a random ideas note:

I can definitely make the update (and download if the current method is kept) command default to your OS (as opposed to Windows), would that be desirable ?

And I can probably make the language default to the language your system locale is using as well. This is likely to be less desirable since many installers are only provided in English. But thoughts ? One of mine here is that for languages other than English I could make the default the system langauge then English if the specified Language doesn't exist (this would require me to implement an additional filter type (the first of this list that exists) that I'd then expose for this and OS and would consequently take a bit longer to do).


avatar
timmy010: this looks useful but i don't know what python is. some background may be useful
https://www.python.org/

It's a scrpting / programming language that compiles the source as it is invoked (similar to Java).

avatar
Kalanyr: The 2nd is GOG side but not really an error as such, it's an item promised as part of Kickstarter that hasn't been provided to GOG (usually such things have ended up on Backerkit or the companies own websites, some of them just haven't happened).
avatar
SirPrimalform: Huh, that's strange. If GOG never had it, how does gogrepo pick up on it? I guess they created a database entry in anticipation or something.
Correct,They show up on the game pages too, usually marked as coming soon , some of them have been coming soon for about 5 years at this point.
Post edited March 02, 2018 by Kalanyr
avatar
Kalanyr: Okay quick straw poll:

Do people prefer download to grab everything in the manifest or have the same defaults as update ?

This is one of those things I've been thinking about but should settle on before I redo the readme. The current way breaks backward compatability with the original in a way that's strictly unnecessary (it just required more work on my part than I was willing to do at the time) whereas I have generally mantained backwards compatability as much as possible. And I think I've worked out most of the required changes while thinking it over. On the other hand reverting to the old way breaks compatability with scripts people have setup since (though I suspect only for a tiny proportion of the user base who download non-English / non-Windows builds).

I'm leaning towards defaulting to the entire manifest (but keeping the filters available of course).

[kohlrak@kohlrak-server ~]$ cat gogupdate.sh
cd gog
./gogrepo.py update -os windows linux -lang en jp ru -updateonly
./gogrepo.py download
./gogrepo.py update -os windows linux -lang en jp ru -skipknown
./gogrepo.py download
Not sure if my script is broken or not, since i have't checked in a while, but i tend to run that, and occasionally delete the manifest and run it for the games that don't get properly flagged as updates.

I have a preference for original language versions for when i can understand the original language. Learning Russian is on my todo list so i have that flagged for now. I natively speak english, and i'm self-taught in japanese in spanish, but spanish is too rare to bother singling out on gog, plus spanish and english translate rather well, anyway.
Post edited March 02, 2018 by kohlrak
avatar
Kalanyr: Okay quick straw poll:

Do people prefer download to grab everything in the manifest or have the same defaults as update ?

This is one of those things I've been thinking about but should settle on before I redo the readme. The current way breaks backward compatability with the original in a way that's strictly unnecessary (it just required more work on my part than I was willing to do at the time) whereas I have generally mantained backwards compatability as much as possible. And I think I've worked out most of the required changes while thinking it over. On the other hand reverting to the old way breaks compatability with scripts people have setup since (though I suspect only for a tiny proportion of the user base who download non-English / non-Windows builds).

I'm leaning towards defaulting to the entire manifest (but keeping the filters available of course).
avatar
kohlrak:

[kohlrak@kohlrak-server ~]$ cat gogupdate.sh
cd gog
./gogrepo.py update -os windows linux -lang en jp ru -updateonly
./gogrepo.py download
./gogrepo.py update -os windows linux -lang en jp ru -skipknown
./gogrepo.py download
avatar
kohlrak: Not sure if my script is broken or not, since i have't checked in a while, but i tend to run that, and occasionally delete the manifest and run it for the games that don't get properly flagged as updates.
That script doesn't actually download the linux or jp / ru versions because download defaults to windows / english (a change I made which is different to the original version, this is why I generally tried to keep backwards compatability)

You don't actually need to delete the manifest either, leaving off updateonly and skipknown will update everything from your account whether or not it's currently in your manifest (and it cleans up stuff that shouldn't be in your manifest automatically).
Post edited March 02, 2018 by Kalanyr
avatar
kohlrak: Not sure if my script is broken or not, since i have't checked in a while, but i tend to run that, and occasionally delete the manifest and run it for the games that don't get properly flagged as updates.
avatar
Kalanyr: That script doesn't actually download the linux or jp / ru versions because download defaults to windows / english (a change I made which is different to the original version, this is why I generally tried to keep backwards compatability)

You don't actually need to delete the manifest either, leaving off updateonly and skipknown will update everything from your account whether or not it's currently in your manifest (and it cleans up stuff that shouldn't be in your manifest automatically).
Well, due to the lack of an internal update script (now you'll see why it's requested), and being unable to constantly follow this topic, i don't update that often. The last version i used before the one i use now, I had to delete the manifest.

So how would I go about getting the versions i'm trying to hit with this version, or is that currently broken entirely? And what would it entail to get it working again? I know C++, assembly (atmel, arm, x86), java, etc, but python isn't on that list, so i'm not sure how to proceed in understanding the problems.
avatar
Kalanyr: That script doesn't actually download the linux or jp / ru versions because download defaults to windows / english (a change I made which is different to the original version, this is why I generally tried to keep backwards compatability)

You don't actually need to delete the manifest either, leaving off updateonly and skipknown will update everything from your account whether or not it's currently in your manifest (and it cleans up stuff that shouldn't be in your manifest automatically).
avatar
kohlrak: Well, due to the lack of an internal update script (now you'll see why it's requested), and being unable to constantly follow this topic, i don't update that often. The last version i used before the one i use now, I had to delete the manifest.

So how would I go about getting the versions i'm trying to hit with this version, or is that currently broken entirely? And what would it entail to get it working again? I know C++, assembly (atmel, arm, x86), java, etc, but python isn't on that list, so i'm not sure how to proceed in understanding the problems.
You want to get gogrepo from here https://github.com/Kalanyr/gogrepo

The most efficient way to do what your current script is doing is
./gogrepo update -os windows linux -lang en jp ru -updateonly
./gogrepo update -os windows linux -lang en jp ru -skipknown
./gorepo clean
./gogrepo.py download -os windows linux -lang en jp ru

I would recommend adding a
./gogrepo verify -os windows linux -lang en jp ru -clean
to the end but be aware that it will be very slow the first time you run it (and I'd recommend doing the bit I'll cover next first by preference in case there is old stuff in your manifest)

And to do a full manifest update
./gogrop update -os windows linux -lang en jp ru
./gorepo clean
./gogrepo.py download -os windows linux -lang en jp ru

And again I'd recommend adding
./gogrepo verify -os windows linux -lang en jp ru -clean
to the end (but again this will take a long time the first time it's run and be much quicker thereafter, this is because it verifies everything the first time it's run but only new stuff after that. If you've got a large collection (~1500 games) it can take ~2 days to verify everything on a first pass but only about a quarter of an hour to veify weekly full manifest updates later on ).

You'll also now see an !orphaned folder in the folder you run the script in, this will contain any outdated files or files that failed to verify, so you can delete them at your leisure or fish them back out in case GOG goofed with something (on rare occassions GOG puts up the wrong MD5 or removes an extra without replacing).
Post edited March 02, 2018 by Kalanyr
avatar
kohlrak: Well, due to the lack of an internal update script (now you'll see why it's requested), and being unable to constantly follow this topic, i don't update that often. The last version i used before the one i use now, I had to delete the manifest.

So how would I go about getting the versions i'm trying to hit with this version, or is that currently broken entirely? And what would it entail to get it working again? I know C++, assembly (atmel, arm, x86), java, etc, but python isn't on that list, so i'm not sure how to proceed in understanding the problems.
avatar
Kalanyr: You want to get gogrepo from here https://github.com/Kalanyr/gogrepo

The most efficient way to do what your current script is doing is
./gogrepo update -os windows linux -lang en jp ru -updateonly
./gogrepo update -os windows linux -lang en jp ru -skipknown
./gorepo clean
./gogrepo.py download -os windows linux -lang en jp ru

I would recommend adding a
./gogrepo verify -os windows linux -lang en jp ru -clean
to the end but be aware that it will be very slow the first time you run it (and I'd recommend doing the bit I'll cover next first by preference in case there is old stuff in your manifest)

And to do a full manifest update
./gogrop update -os windows linux -lang en jp ru
./gorepo clean

And again I'd recommend adding
./gogrepo verify -os windows linux -lang en jp ru -clean
to the end (but again this will take a long time the first time it's run and be much quicker thereafter).
Weird, what exactly am i doing wrong right now? Outside of the verify, and the plain-jane update without options, where am i failing?
avatar
kohlrak: Outside of the verify, and the plain-jane update without options, where am i failing?
"gogrepo.py download" is equivalent to "gogrepo.py download -os windows -lang en", which means it skips the Linux, Japanese and Russian languages.
The "gogrepo.py clean" is just so it moves files it doesn't recognize (aka older ones) to the !orphaned directory for later review.
avatar
Kalanyr: You want to get gogrepo from here https://github.com/Kalanyr/gogrepo

The most efficient way to do what your current script is doing is
./gogrepo update -os windows linux -lang en jp ru -updateonly
./gogrepo update -os windows linux -lang en jp ru -skipknown
./gorepo clean
./gogrepo.py download -os windows linux -lang en jp ru

I would recommend adding a
./gogrepo verify -os windows linux -lang en jp ru -clean
to the end but be aware that it will be very slow the first time you run it (and I'd recommend doing the bit I'll cover next first by preference in case there is old stuff in your manifest)

And to do a full manifest update
./gogrop update -os windows linux -lang en jp ru
./gorepo clean

And again I'd recommend adding
./gogrepo verify -os windows linux -lang en jp ru -clean
to the end (but again this will take a long time the first time it's run and be much quicker thereafter).
avatar
kohlrak: Weird, what exactly am i doing wrong right now? Outside of the verify, and the plain-jane update without options, where am i failing?
Sorry, was still editing my above post. I've added a paragraph at the end and fixed the plane jane update to actually download stuff. The bit where your script would fail on the current version was the lack of "-os windows linux -lang en jp ru " on the "./gogrepo download" as that command currently defaults to windows / english, if you're still using wooly's build or a very very early version of mine, it would still work though, as I only broke backwards compatability when I was improving filtering on downloads (I can restore it , but I was adding a lot of the stuff at the time and couldn't be bothered).
Post edited March 02, 2018 by Kalanyr
avatar
kohlrak: Weird, what exactly am i doing wrong right now? Outside of the verify, and the plain-jane update without options, where am i failing?
avatar
Kalanyr: Sorry, was still editing my above post. I've added a paragraph at the end and fixed the plane jane update to actually download stuff. The bit where your script would fail on the current version was the lack of "-os windows linux -lang en jp ru " on the "./gogrepo download" as that command currently defaults to windows / english, if you're still using wooly's build or a very very early version of mine, it would still work though, as I broke backwards compatability when I was improving filtering on downloads.
So, it's actually pointless to have it even in the updates list, too, then, right? Or is it necessary throughout the whole chain (just curiosity, since i figure the extra text on the line wouldn't matter). And the -skipknown update line wouldn't cancel our erase the results from the -updateonly line?
[kohlrak@kohlrak-server ~]$ ./gogupdate.sh
./gogupdate.sh: ./gogrepo.py: python3: bad interpreter: No such file or directory
./gogupdate.sh: ./gogrepo.py: python3: bad interpreter: No such file or directory
./gogupdate.sh: ./gogrepo.py: python3: bad interpreter: No such file or directory
./gogupdate.sh: ./gogrepo.py: python3: bad interpreter: No such file or directory
./gogupdate.sh: ./gogrepo.py: python3: bad interpreter: No such file or directory
[kohlrak@kohlrak-server ~]$ python --version
Python 2.7.14
[kohlrak@kohlrak-server ~]$ python3
Python 3.6.4 (default, Feb 8 2018, 14:53:12)
I think manually changing it in the file myself should work for now, but not sure for the long term.
Post edited March 02, 2018 by kohlrak
avatar
Kalanyr: Sorry, was still editing my above post. I've added a paragraph at the end and fixed the plane jane update to actually download stuff. The bit where your script would fail on the current version was the lack of "-os windows linux -lang en jp ru " on the "./gogrepo download" as that command currently defaults to windows / english, if you're still using wooly's build or a very very early version of mine, it would still work though, as I broke backwards compatability when I was improving filtering on downloads.
avatar
kohlrak: So, it's actually pointless to have it even in the updates list, too, then, right? Or is it necessary throughout the whole chain (just curiosity, since i figure the extra text on the line wouldn't matter). And the -skipknown update line wouldn't cancel our erase the results from the -updateonly line?
The update / download -os and -lang switch control different things:
On update it controls what is looked at on GOG.com and then written into your manifest
On download it controls whatt from within your manifest is downloaded from GOG.com

This is useful for eg people who use the same manifest but download windows / linux installers and extras to different directories or such.

Correct, your manifest is "living" , only the parts that change on GOG.com (within your filters) are updated by update.


[kohlrak@kohlrak-server ~]$ ./gogupdate.sh
./gogupdate.sh: ./gogrepo.py: python3: bad interpreter: No such file or directory
./gogupdate.sh: ./gogrepo.py: python3: bad interpreter: No such file or directory
./gogupdate.sh: ./gogrepo.py: python3: bad interpreter: No such file or directory
./gogupdate.sh: ./gogrepo.py: python3: bad interpreter: No such file or directory
./gogupdate.sh: ./gogrepo.py: python3: bad interpreter: No such file or directory
[kohlrak@kohlrak-server ~]$ python --version
Python 2.7.14
[kohlrak@kohlrak-server ~]$ python3
Python 3.6.4 (default, Feb 8 2018, 14:53:12)
avatar
kohlrak: I think manually changing it in the file myself should work for now, but not sure for the long term.
Woops, that's my bad, the first line is a directive that's necessary on my PC because I have python 2 and python 3 installed and set to default to python 2, for compatability with some old work scripts and I don't want it to use python 2 by default for this script (since python 2 will be EOL soon). I should have removed it before I uploaded it but didn't think about it. You can safely delete it. I'll try and remember to axe it in future builds.
Post edited March 02, 2018 by Kalanyr
avatar
kohlrak: So, it's actually pointless to have it even in the updates list, too, then, right? Or is it necessary throughout the whole chain (just curiosity, since i figure the extra text on the line wouldn't matter). And the -skipknown update line wouldn't cancel our erase the results from the -updateonly line?
avatar
Kalanyr: The update / download -os and -lang switch control different things:
On update it controls what is looked at on GOG.com and then written into your manifest
On download it controls whatt from within your manifest is downloaded from GOG.com

This is useful for eg people who use the same manifest but download windows / linux installers and extras to different directories or such.

Correct, your manifest is "living" , only the parts that change on GOG.com (within your filters) are updated by update.
Gotcha.

I solved the issue (bad interpreter) in my last post by updating the first line to #!/usr/bin/python3, but now i have a new issue. Seems to be working otherwise, but, wow.

EDIT: nvm, read the actual error.
[kohlrak@kohlrak-server ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 997M 0 997M 0% /dev
tmpfs 1006M 0 1006M 0% /dev/shm
tmpfs 1006M 680K 1005M 1% /run
tmpfs 1006M 0 1006M 0% /sys/fs/cgroup
/dev/mapper/vg_kohlrakserver-lv_root 50G 11G 36G 23% /
tmpfs 1006M 24K 1006M 1% /tmp
/dev/sda1 477M 134M 314M 30% /boot
/dev/mapper/vg_kohlrakserver-lv_home 1.8T 1.7T 0 100% /home
tmpfs 202M 0 202M 0% /run/user/1000
[kohlrak@kohlrak-server ~]$
Attachments:
Post edited March 02, 2018 by kohlrak