It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
avatar
Kalanyr: Yes, you can store the (pseudo)-OAuth session data , which is pretty much the following.

{'access_token': 'REDACTED',
'expires_in': 3600,
'expiry': REDACTED,
'refresh_token': 'REDACTED',
'scope': '',
'session_id': 'REDACTED',
'token_type': 'bearer',
'user_id': 'REDACTED'}

You can then use it whenever you need to login again. You do need to do a session renew request as documented when it expires (which is each hour), IIRC you have to calculate the actual expiry time yourself (which is why I have an expiry field as well as the expires_in. but you could keep acquisition time + expires_in and use that instead. )

(If you want to send me a PM , I can send you the still a mess code for the next version of GOGrepoc but it does actually have fully working GOG OAuth code you can copy)
avatar
Geralt_of_Rivia: First and foremost, thanks for answering my questions. I hope I don't take up too much of your time.

Whenever? Even after a day or longer? The API doc gave me the impression that the whole OAUTH data expires after an hour which means that the refresh_token also becomes invalid. Or in other words that the refresh had to be done within the hour of validity to get new OAUTH data before the old expires.

If that is not the case and the refresh_token stays valid even after expiration of the access_token then I can skip the login if I have an old token that I can use to get a new one.

But that leads to several new questions: Do you know how long the refresh_token stays valid? Days, weeks, months? If I try to use the old refresh token to get new OAUTH data how do I know if that succeeded? Is the result I get on a failed attempt not a valid JSON file? Or do I have to check the HTTP status code (401 or 403)? Does an attempt to refresh OAUTH with an invalid or expired refresh_token put me on the blacklist?

Thanks for offering code but that's not really neccessary. The problem isn't producing code. The problem is coming up with a valid program logic when the server's behaviour isn't fully documented and the data the server returns is woefully incomplete.
Yes, it's indefinite as far as I can tell, the only thing an explred session can do is use the refresh token to request a new token,. The return from a successful refresh is a new token (same as a login, so it includes a new refresh_token , etc) so you can verify it the same way. It shouldn't blacklist you as long as you don't spam invalid refresh attempts.
avatar
Geralt_of_Rivia: But that leads to several new questions: Do you know how long the refresh_token stays valid? Days, weeks, months? If I try to use the old refresh token to get new OAUTH data how do I know if that succeeded? Is the result I get on a failed attempt not a valid JSON file? Or do I have to check the HTTP status code (401 or 403)? Does an attempt to refresh OAUTH with an invalid or expired refresh_token put me on the blacklist?
avatar
Kalanyr: Yes, it's indefinite as far as I can tell, the only thing an explred session can do is use the refresh token to request a new token,. The return from a successful refresh is a new token (same as a login, so it includes a new refresh_token , etc) so you can verify it the same way. It shouldn't blacklist you as long as you don't spam invalid refresh attempts.
I see. I never reused the OAUTH data but did a new login on every program start. I'll have to try and change that. That will hopefully keep me off the blacklist once I'm off again. :-(
avatar
Kalanyr: This depends what you mean by "fixed", there was a period during the early COVID times, where they weren't marking things as updated at all, they are now, so that's "fixed". And they are presently a *lot* better about marking things that are updated as updated than they were in the past.

The main problem is GOG does "administrative" updates which don't change anything from a user perspective and are mainly about cleaning up internal stuff, so you don't really want that marked as updated because redownloading it is a waste. Which means it's hard to tell when they've let something slip and when it's just an administrative thing.

I guess you could setup a script that runs a gogrepoc new/updated update, then a full update afterwards, notes down what's changed in the second one, and then do stuff like compare the file names / versions to see what's a "real" change that was missed.
Ah, that's kind of irritating. I guess I'll just go back to using the new/updated mode frequently and then redo the manifest from scratch every few months to make sure nothing has slipped through the net.
I've now gotten around to setting up things for a download speed comparison test between original 'gogrepo.py' and the forked version.

And unfortunately, it turned out like I suspected.

I used a separate program folder and copy of my GUI for each.
Both were set to use 1 thread.
Both had the same destination folder.
Each used their own copy of the manifest, that each created from scratch themself and had updated.

The original version consistently downloaded at around 4 MB/s.
Whereas the forked version was consistently under 1 MB/s for the exact same game file. It often seems to almost lose the connection, and sometimes does.

So it seems pretty clear that the forked version is extremely slower and not reliable ... at least for me. :-(

Maybe this has something to do with the 'Resume' feature of the forked version or the fork version just downloads differently.

I did the original download first to completion.
Then a few minutes later I followed with the forked version download. I eventually gave up and cancelled the download after many minutes.

EDIT
After doing this post, I went back and started redoing with original gogrepo.py, and got the same speeds, speeds I can never get with the forked version or anywhere near it.
Post edited August 22, 2020 by Timboli
avatar
Timboli: After doing this post, I went back and started redoing with original gogrepo.py, and got the same speeds, speeds I can never get with the forked version or anywhere near it.
Weird, I recall that my download speed with Kalanyr's stable version a pair of months ago was like 13mbps x 4..
avatar
phaolo: Weird, I recall that my download speed with Kalanyr's stable version a pair of months ago was like 13mbps x 4..
I suspect it may be OS &/or Processor maybe even RAM related.
It's quite probably my processor just isn't keeping up with all that Kalanyr's code is doing ... or there is some kind of optimization happening that isn't available in Win 7.
All a guess by me, but it is clear his code is definitely different to that used by the original gogrepo.py.
avatar
phaolo: Weird, I recall that my download speed with Kalanyr's stable version a pair of months ago was like 13mbps x 4..
avatar
Timboli: I suspect it may be OS &/or Processor maybe even RAM related.
It's quite probably my processor just isn't keeping up with all that Kalanyr's code is doing ... or there is some kind of optimization happening that isn't available in Win 7.
All a guess by me, but it is clear his code is definitely different to that used by the original gogrepo.py.
Wait, you're using Win7? I'm using that too lol.
I also have a quad core i5, with 16 gb ram.
Is your normal download speed higher?
avatar
Timboli: I've now gotten around to setting up things for a download speed comparison test between original 'gogrepo.py' and the forked version.

And unfortunately, it turned out like I suspected.

I used a separate program folder and copy of my GUI for each.
Both were set to use 1 thread.
Both had the same destination folder.
Each used their own copy of the manifest, that each created from scratch themself and had updated.

The original version consistently downloaded at around 4 MB/s.
Whereas the forked version was consistently under 1 MB/s for the exact same game file. It often seems to almost lose the connection, and sometimes does.

So it seems pretty clear that the forked version is extremely slower and not reliable ... at least for me. :-(

Maybe this has something to do with the 'Resume' feature of the forked version or the fork version just downloads differently.

I did the original download first to completion.
Then a few minutes later I followed with the forked version download. I eventually gave up and cancelled the download after many minutes.

EDIT
After doing this post, I went back and started redoing with original gogrepo.py, and got the same speeds, speeds I can never get with the forked version or anywhere near it.
Yes, my code is a little heavier because it monitors the download threads and kills/restarts them if they've been unresponsive for too long. This is mainly necessary to stop soft locks when all the threads failed (it also reduces mysterious download failures where a thread went AWOL significantly).

This really shouldn't have a particularly significant effect on most CPUs but I'm by no means an expert on how good pythons threading / timer code is for stuff like this under the hood across OS/architectures.
avatar
phaolo: Is your normal download speed higher?
Yes indeed, more than four times the speed when using the original gogrepo.py. Not only that, but it is a fairly stable speed, whereas with the forked version, the speed is all over the place and often showing '0' momentarily much of the time, and even on occasion losing the connection, and then needing to resume. It is so slow on average that it is painfully slow.
avatar
Kalanyr: Yes, my code is a little heavier because it monitors the download threads and kills/restarts them if they've been unresponsive for too long. This is mainly necessary to stop soft locks when all the threads failed (it also reduces mysterious download failures where a thread went AWOL significantly).

This really shouldn't have a particularly significant effect on most CPUs but I'm by no means an expert on how good pythons threading / timer code is for stuff like this under the hood across OS/architectures.
The short answer to that I guess, is that both the original gogrepo.py and your forked version are using the same version of Python, but clearly in a different manner.

My processor in this PC is low powered compared to the norm these days, and only dual core, so I suspect it cannot keep up with the number crunching your code is doing. NOTE - I don't use this PC for gaming, it is purely for my web use (downloading, email, etc) and a bit of coding.

I am guessing that even when I select '-resume no resume' that your download code is still more memory intensive than what is used by the original gogrepo.py? It would be nice if I could gain the other benefits of your forked version without that overhead. Personally, I have a fairly stable web connection, so it would be a rare moment for me to need the resume option.
Post edited August 24, 2020 by Timboli
avatar
Kalanyr: Yes, my code is a little heavier because it monitors the download threads and kills/restarts them if they've been unresponsive for too long. This is mainly necessary to stop soft locks when all the threads failed (it also reduces mysterious download failures where a thread went AWOL significantly).

This really shouldn't have a particularly significant effect on most CPUs but I'm by no means an expert on how good pythons threading / timer code is for stuff like this under the hood across OS/architectures.
avatar
Timboli: The short answer to that I guess, is that both the original gogrepo.py and your forked version are using the same version of Python, but clearly in a different manner.

My processor in this PC is low powered compared to the norm these days, and only dual core, so I suspect it cannot keep up with the number crunching your code is doing. NOTE - I don't use this PC for gaming, it is purely for my web use (downloading, email, etc) and a bit of coding.

I am guessing that even when I select '-resume no resume' that your download code is still more memory intensive than what is used by the original gogrepo.py? It would be nice if I could gain the other benefits of your forked version without that overhead. Personally, I have a fairly stable web connection, so it would be a rare moment for me to need the resume option.
The resume code that controls is unrelated (it's for resuming manifest updates). The download resume shouldn't have any real effect unless your system struggles computing MD5 hashes on small chunks but that seems unlikely given the computing power when MD5 was introduced.

One other person has mentioned this issue before so it may be an issue with the Python Timer in some specific environment.

Have you tried using whichever of Python 2 / Python 3 you aren't currently using ? Particularly if you're using Python 2.
avatar
Timboli: My processor in this PC is low powered compared to the norm these days, and only dual core, so I suspect it cannot keep up with the number crunching your code is doing. NOTE - I don't use this PC for gaming, it is purely for my web use (downloading, email, etc) and a bit of coding.
So what does the task manager say about the CPU usage? How much of the CPU power is used (by gogrepo/python, and overall)?

I don't recall seeing any actual speed issues running gogrepo on e.g. the low power Raspberry Pi4 which, I think, has quite "weak" CPU compared to modern PCs, even if it has 4 cores. Then again if this is somehow related to Windows (7), obviously RPi4 is not running that, but a 32bit Linux.

I might try it on one of my older laptops as well, e.g. with an Intel Pentium P6200 CPU, which has only two cores:

https://ark.intel.com/content/www/us/en/ark/products/50176/intel-pentium-processor-p6200-3m-cache-2-13-ghz.html

Then again I don't have Windows 7 installed on that laptop anymore, it has Windows XP and 64 bit Linux Mint. Not sure if python is available for Windows XP, or whether running gogrepo on the Linux partition is an invalid comparison, but still it would show if that kind of CPU has any performance issues with gogrepo.
I have a question for all users of gogrepo.

I have heard that doing a full manifest update or creating it from scratch takes a very long time. So long that some people even let it run over night.

So the question here is: How long does a full manifest update take on your system?

Since that obviously not only depends on your system but also on the size of your library please also state how big your library is.

Perfekt information would be: "xxx games, yyyy files, full manifest update takes h:m:s"

But I would also be happy with information that is less than perfect (e.g. you don't know the number of files of all your games or if you have never checked how long gogrepo takes exactly but only know 'it takes approximately two and a half hours').

The reason I'm asking is that I'm currently writing a downloader myself completely from scratch. And if the data collection takes a very long time I'd like to know how long I can expect it to run. If my program takes significantly longer than gogrepo I'd have to think about optimization.

That's why I need values for comparison.
avatar
Geralt_of_Rivia: So the question here is: How long does a full manifest update take on your system?
I posted the following about a month ago.

All up, for 978 games in my GOG library, it took 5 hours and 50 minutes to get all the required detail via UPDATE ALL to build the manifest file on my PC. This was for both Windows and Linux plus all game extras.

--------------------------------------------------

Don't know if you are aware, but I have been working on a Windows GUI for gogrepo.py, called GOGRepo GUI.

I am about to release the finished latest version.

One of the options my GUI provides, is a two method feature called - Update In Stages.

Method 1 builds a manifest from scratch, for the number of games specified. A CONTINUE button allows you to add more of your games, once again for the number you specify. You repeat with the CONTINUE button until done. This all means you can build up your manifest, with a chunk of free time here or there, over days even if you want.

Method 2 removes and then replaces the specified number of games from your manifest, creating a completely new entry for each of those games, it also keeps a comparison file, if changes detected. Once again, you use the CONTINUE button to work through all your games at your leisure.

I haven't created a separate thread (topic) for my program yet, so it shares a thread with two others, the main one being GOGPlus Download Checker, which is what the thread is named after.

My latest post is here - https://www.gog.com/forum/general/gogplus_download_checker/post144
Post edited August 25, 2020 by Timboli
avatar
Kalanyr: Have you tried using whichever of Python 2 / Python 3 you aren't currently using ? Particularly if you're using Python 2.
I had Python 3.4.1 installed at the time I first started using the original gogrepo.py.
When I started using your forked version, I found I had to update to get the other libraries required, so I updated to Python 3.7.0.

I am not that familiar with Python and its requirements, but it seems unlikely that installing a version 2 variant would work. when I had to upgrade from version 3.4.1 to get those other required libraries ... especially pyopenssl.

So Python 3.7.0. is what I currently use for gogrepo.py, original or forked.

The original works as well with Python 3.7.0 as it did with Python 3.4.1 ... I get full download speed.
avatar
timppu: So what does the task manager say about the CPU usage? How much of the CPU power is used (by gogrepo/python, and overall)?
When I get some time, I might have a go at checking that, just for double confirmation, but it isn't going to solve anything ... probably only tell me what I feel I already suspect.
Post edited August 25, 2020 by Timboli