I regularly use gogcli for my backups and it works fine. However, I don't often use the full manifest generation to validate all the games. I believe I have the specific game that trips you up, so I'll give it a try this weekend.

Otherwise, concerning things that may pop up:

Occasionally, it can happen that gog has a bad file (the unintentional kind of bad file that you need to open a support ticket and get them to fix). When that happens, if its an important file (ex: an installer), I'll open up a support ticket with GOG, but if the file is not important (ex: a patch file), I'll usually just add an exception for it in my manifest and gogcli will ignore it.

Besides the above, GOG's file metadata endpoints (used to retrieve the expected checksum of the file) can sometimes be bad (it used to be a lot worse, I think they've improved it) and from what you describe, I believe that is what is happening there. When that happens, gogcli will retry about 4 times or so and then if it still fails, it will circumvent the metadata endpoint by simply doing a mock download of the file and computing the checksum itself.

If the later situation happens, there might be a delay (especially for a larger file, downloading it may take some time depending on the file's size and the speed of your internet connection) while the file is downloaded and the checksum is computed. when that happens, after the delay, you'll get a message informing you that the longer workaround method has been used for the file.

Do note that the above workaround solution that been implemented only in more recent versions of gogcli so if your version lag behinds, then you'll just get an error as there won't be any workarounds for you. Given my limited personal time, I can only commit to fixing problems in the latest release.

Hi, Magnitus. Thanks for replying.

I should've pointed out that I don't really think that gogcli is really the source of issue, only that maybe it just doesn't cope with whatever is issue on my side (maybe some Windows update broke something related to this, because that is really the only thing that changed from January 2022 to November 2022 on my PC).

I'll wait for your research, the rest of the message is just to elaborate further on what I did try.

Just to make this more visible:
Versions I tried with:
GOGcli GUI v2.8, v3.4, v3.5, v3.7, combined with gogcli v0.17.2 (first version I ever used, way back when there were no such issues), v0.20 and v0.21.

I was actually aware something is up on my side since neither Timbolis or your thread have posts like mine. I'm really hoping you or anyone can reproduce the bug and it is something fixable on my end. One of the outcomes could be a conclusion that the issue is on GOG side, which would suck.

Got it. On the same line, I think I have a completed manifest from January 2022 (I had e.g. 140, Witcher 3 at the time, but not 7.62 games), would that be of any use for maybe forcing manifest finishing?

I didn't mention it, but I've seen all of those additional messages - workaround worked in some cases, that is for sure. From how I remember it, I left it running for Witcher 3 for 15-20 minutes. I don't know if it failed by itself, but most likely I closed the terminal by hand.

Today I made a new directory, downloaded again and unzipped newest gogcli and GUI. I initiated manifest for The Witcher 3: Wild Hunt - Complete Edition, here is what is shown in terminal:

The Witcher 3: Wild Hunt - Complete Edition
[manifest writer] Generating/Updating manifest for 1 games
[sdk] getDownloadFileInfo(downloadPath=/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/92788) -> body retrieval error: stream error: stream ID 177; PROTOCOL_ERROR; received from peer. Will retry.
[sdk] getDownloadFileInfo(downloadPath=/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/92788) -> body retrieval error: stream error: stream ID 1; PROTOCOL_ERROR; received from peer. Will retry.
[sdk] getDownloadFileInfo(downloadPath=/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/92788) -> body retrieval error: stream error: stream ID 1; PROTOCOL_ERROR; received from peer. Will retry.
[sdk] getDownloadFileInfo(downloadPath=/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/92788) -> body retrieval error: unexpected EOF. Will retry.
[sdk] getDownloadFileInfo(downloadPath=/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/92788) -> body retrieval error: stream error: stream ID 3; PROTOCOL_ERROR; received from peer. Will retry.
[sdk] Bad metadata for /downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/92788: File metadata was still fetched using much longer workaround method.
[sdk] getDownloadFileInfo(downloadPath=/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/92797) -> body retrieval error: unexpected EOF. Will retry.
[sdk] getDownloadFileInfo(downloadPath=/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/92797) -> body retrieval error: unexpected EOF. Will retry.
[sdk] getDownloadFileInfo(downloadPath=/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/92797) -> body retrieval error: unexpected EOF. Will retry.
[sdk] getDownloadFileInfo(downloadPath=/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/92797) -> body retrieval error: unexpected EOF. Will retry.
[sdk] getDownloadFileInfo(downloadPath=/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/92797) -> body retrieval error: unexpected EOF. Will retry.

After this 5th retry no further operation occurred for exactly 5 minutes, and then it continued, with the same 5 retries. 15 minutes later, only the terminal cursor is blinking, no new outputs.

Starting the "Manifest for all" at the previous list entry (which is The Witcher 2, and that seems to be added to manifest with no messages) produces the same errors, but for quite longer and doesn't seem to stop.
It switches between:
- [sdk] getDownloadFileInfo(downloadPath=/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/92812) -> body retrieval error: stream error: stream ID 3; PROTOCOL_ERROR; received from peer. Will retry.
- [sdk] getDownloadFileInfo(downloadPath=/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/92797) -> body retrieval error: unexpected EOF. Will retry.

with workaround message popping up sometimes.

Couple of things I can think of:

1. I pretty much always run my tool in Ubuntu. I do have a Windows 10 box and I've troubleshooted the tool on it before for someone else, but not the latest iteration so I'll do that just in case. I'm assuming you are on Windows 10?

2. Not that it should make a great difference beyond wait time, but what is the speed of your internet connection? Mine is about 50mbs. I know that GOG's servers have been known to give up on large files when throughput is too low and it takes forever. If the speed of your internet connection if well below mine and you have some large files, consider reducing the concurrency (which defaults to 4). Its usually the -r or --concurrency flag (if you feel things are bogged down, you can set it to 1)

3. Just to be sure I understand your situation correctly, when you get those errors, does the tool stop without completing or does it seems like it is taking a long time and you forcefully terminate it? As previously mentioned, there is a default concurrency of 4, meaning that while the tool is performing the workaround on one file on one of its thread, it might be processing other files on some of its other threads and displaying that on the output concurrently. That's normal and fine as long as it doesn't abruptly stop without completing.

4. If ever you want to get A LOT of details from the sdk (probably way more than you bargained for), you can always set the log level to debug (-g or --log-level flag).

Otherwise, unrelated to the troubleshooting of the tool given what you are indicating, if ever get the impression that your OS installation is not quite right, I would strongly recommend that you backup what is important, then format your hard drive and reinstall your OS. Doubts about that will taint everything you do with your computer (I know the feeling, been there) and you want to nip that in the bud.

It is a good task to perform once in a while for the sanity of your system (OSes are not perfect and their operations can degrade over time, especially on a user desktop which tends to be a more fluid environment where a lot of poorly controlled things are happening). I do that myself once every 1-2 years at least.

No worries, and I am going to have to plead a similar thing, as I am about to go to bed, and then I will be leaving on a road trip in the morning and be away for about two weeks.

I haven't got time to properly analyse what is going on now, but will leave it in Magnitus' very capable hands, and if still needed also look into it myself on my return.

Hope you get it solved.
Cheers!

fyi, I generated an entire manifest with Linux and no roadblocks.

I added an exception for 3 patches that were missing in the manifest generation progress file (I should probably automate that part), but that's it.

I'll generate one on Windows after the spring sale is over. For now, I'll give GOG's servers are break as it seems like they will be stressed for a bit.

First point, I am confused to report (like "happy to report", but with more "what gives") that now, even with errors, the manifest creation for the whole list is advancing instead of looping errors and retries for 10-20 minutes.

The only thing I did, because of an random idea, was to manually delete manifest.txt and then start "Download all" in the GUI. I have no idea what I'm doi... I mean, if deleting the manifest helped anything.

(originally was a lot longer section) Second point, the answers:
- Windows 10 (I agree on the "time to reinstall" point, just that I haven't been having any issues with Win that could be related to gog),
- internet connection is close to 100 Mbps (concurrency reductions sounds good though, will try).

In multiple sessions I tried leaving the tool to finish by itself, but I would come back to the terminal for half an hour and see no new errors, and no change to the manifest .json or txt file.

I have seen that some things were added to manifest, and I realized that successful additions just aren't shown in terminal, so I figured to let it run for what I think is rational amount of time, before stopping manually. But that didn't work.

So, earlier behavior: Try downloading/manifest creation -> starts processing problematic game, 140 -> terminal either closes by itself with Retrieval error (rarely happened) or never closes, with no new prints of error for 20 minutes, and doesn't advance to other games.

Current behavior, as it is running while I'm typing this -> starts processing problematic game, 140 -> there was an "will retry" output -> the terminal closes -> 15 days game is being processed.

E.g. Alien Isolation (didn't know this one had issues as well) got so many error and retry messages, but those sets of messages included the names of different DLCs, so I figured there was progress, and held of from closing the terminal.

Here, I believe you were running gogcli with the GUI layer the whole time if I understand correctly?

If you find that you have a lot of issues with that in the future, I encourage you, at least temporarily while you are troubleshooting things, just run gogcli by itself on a terminal and report your findings.

For my part, I solely maintain the gogcli client and I'll be able to provide assistance to the extent that problems can be attributed to a specific command in the client running independently without any interference.

If you find that a problem results from the interaction between gogcli and the GUI, then Timboli would be the person to help you out with that as he's the one maintaining the GUI layer described in this thread.

Could be said like that, since I used gogcli no more than couple of times by itself only as a confirmation of issues for some games I had with GUI. With it I got the same results as with GUI - errors, errors, errors, didn't seem to go anywhere.

I will try what you said, I was a bit lazy by jumping into the GUI solution directly, but the GUI was and is still appealing.

Thanks for all your responses and suggestions.

No worries, the separation between who maintains gogcli and who GUI is quite clear. I only wrote in this thread since I was using GUI, and as a follow up to November message, wanted to continue on and hopefully close the topic.

I left the tool running, no interaction from me, and almost everything seems to be fine now - manifest file is getting populated and it seems everything is there. Timbolis ManifesttoDB app is helping me confirm that.

I won't trouble the thread any longer, as it seems that it whatever was going on just went away after 5 months.

I need to report that one specific game can still stop the process (seems in the GUI only) entirely.

Started the manifest creation from GUI. I came back to the computer (after 6 hours) to see terminal not showing up anymore and "Retrieval failed! Error 3" window, for the game "Kajko i Kokosz" - the issue was present in January 2022 as well, so back then I just continued with the manifest with the game after that one.

gogcli by itself has no issue to make it work.

Command I used: gogcli manifest generate --title="Kajko i Kokosz"
[manifest writer] Generating/Updating manifest for 1 games
[manifest writer] Got all info on game with id 1720224179. 0 games left to process

I uploaded a screenshot from the library, and notice the issue when "i" is inputted. ("i" = "and")

Game can be downloaded normally through the browser. It is delisted, and I guess not a lot of people using the GUI will encounter the issue.

I am back from holidaying now, but it seems you have it mostly sorted.

I'm not sure what I can do about the "i" issue, as it seems language based. But maybe it is related to Unicode.

Hello again. I have finally managed to focus on downloading my library and GogCli GUI finished the job for me. Many thanks to both you and Magnitus.

Used "Download All" functionality all the way. Worked exactly as advertised.

Couple of notes regarding some games:
- Kajko i Kokosz - manifest retrieval and download fail here as before, no changes, and I ignore it.

- REVOLVER360 RE:ACTOR - The ":" between "RE" and "ACTOR", prevented game folder from being created in the target folder - so there was a temp folder with description and changelog txts and all other files were downloaded to target folder instead (all other games with ":" have their folders created with no issues, so the folder not being created must be because ":" doesn't have space after it in RE:ACTOR).
Short solution: created folder by myself and moved files there.

- Riven The Sequel to Myst - Download All failed here like with Kajko i Kokosz, with "Retrieval failed! Error 3". Surprisingly, I found the reason quite fast.
When Download All list was created, game name was saved as "Riven The Sequel to Myst". But in browser, in my library, the game is now called "Riven (1997)". So, in Download All list, I just changed "Riven The Sequel to Myst" -> "Riven (1997)", and it worked.

- Shadow tactics - Blades of shogun - some files have "+" and "&" characters. They are downloaded, but are left in target folder.

Goodo, thanks for the thanks and the feedback, and I am glad it worked mostly well for you.

Every now and then as you have probably realized, a game name in our library is changed at GOG. A regular CHECK or GET GAMES LIST will pick up all those, but you would likely have to retrieve the manifest again for each, which is made harder by not knowing which have been renamed ... though usually they can be found by using the LAST button to cycle through Recent Additions ... you can also check that text file. Another method is to use the Check For Orphan Entries option in the right-click Games List menu option Manifest. As that will reveal what games are no longer listed via their old name.

And yes, there is always some character issue I have overlooked, and generally you don't expect a colon to exist mid word of a name. If I remember, I'll add a fix to the next update and also look into the other characters you mentioned.

No GogCli and GUI issues from what I've witnessed, so positive feedback it is.
One thing I can mention that required attention from me: "Done" mark in DownAll.txt

If "Cancel" checkbox is marked during the download of one game with "Download All",
- the download is interrupted after current file has been downloaded, which is excellent, no half-done, corrupted files
- checks of file size and checksum are performed, all is still well
- But, in DownAll.txt, the game is marked as "Done" even though not all files are downloaded.
- At next Download All, that unfinished entry is skipped, and next entry with "0" is downloaded.

Or I missed something, again? I only interrupted downloads when the speed drops to KBs, and I have to leave or something.

Just to clarify, I really didn't mind this, I was glad I payed attention and noticed it early (kind of my line of work), and doing some manual work editing DownAll.txt wasn't problematic, with easy to access GogCli GUI files.

Yeah, I've seen it happened a couple of times. In this case it was just "good timing" for name change to happen some time after my Download All list was created, haha.

It's an artsy-arcade-gamey name, it could be worse probably.

What do you suggest to do after last game in my current Download all list has been downloaded?
- I've bought 10+ games, could download them myself, but
- there are also DLCs and extras for already downloaded games.

I am wondering what is the logical option to use next to download new games and DLCs, if some Database rechecking is needed.

It seems to me that I should NOT create new Download all list, which would force GUI to deal with skipping downloaded files. I imagine that could take a while.

In my current setup, I first let GUI make a manifest of all games, then used Create list for Download All, and used only Download all. No other functions were used, I didn't have time to get into available options, just wanted to start downloading asap. My bad for not researching/reading more, but on the other hand, GUI options are not messed up.

I really couldn't advise you, as it has been so long since I looked at that portion of the code, and I've never done a full Download All myself, as I had all my games before I developed that.

In reality, because you the user instigated a cancel, then as you found, you were aware that not all files for the last game were downloaded, so it isn't like you were kept in the dark. So I suspect it is best dealt with it how you did.
.
From here on out, it is likely about Updates and new game purchases.
You could regularly click the CHECK or GET GAMES LIST button, which will keep the program listing up-to-date with any new games you've purchased, plus game name changes, and the Updated marker. If GOG happen to add something to your library, that should be detected too. This is where clicking the LAST button to work backwards through the recent additions, is very handy, especially with the DOWNLOAD LIST facility, which I mention next.

With an entry marked as Updated (in red), you will need to select and click the ADD TO MANIFEST button, to have that entry replaced in the manifest with the latest file information. That will hopefully mean that only new files appear in the download window.

Yeah, no I wouldn't do that. I see the Download All as a kind of once off thing. If you have kept pace with downloads and purchases, then you should work on an individual basis from now on, taking advantage of the DOWNLOAD LIST facility of holding down CTRL while clicking the DOWNLOAD button. Doing that for each of the games you want to download, so you can download a group of them. That's what I use all the time, if I have more than one game to download.

Here is what I do.

After I buy one or more games at GOG, get a freebie or demo etc.

The first thing I do, is click that CHECK or GET GAMES LIST button.
Once that process has completed and the Games List fully populated, I use the LAST button to cycle through the recent additions to the list. NOTE - To start over with the LAST button, hold down CTRL while clicking the button, that will take the cycle back to the most recent addition.

As I cycle with the LAST button, I click the ADD TO MANIFEST button for each new addition game. And after doing that, if more than one game is going to be downloaded, I click the DOWNLOAD button while holding down CTRL, to add the game to the DOWNLOAD LIST. The first time you click the DOWNLOAD button with CTRL held down, the button is renamed to DOWNLOAD LIST. I just repeat for any other games, remembering to hold down CTRL still for each.

Once all the desired games are on the DOWNLOAD LIST, I click that button without CTRL held down, and I end up, after a dialog or two at the Download Files window. If asked to get latest updates for the manifest, I click cancel, as I have already gotten them using the ADD TO MANIFEST button.

That's more or less it. Fairly quick and easy really.

BY THE WAY
I am yet to upload the last few versions of my GUI to GitHub. GitHub became somewhat of a pain with their new two factor checking method, and so I have been avoiding them for a while. I also use MEGA for my updates, though they have no download counter. I could give you a MEGA link for the latest & greatest via a PM if you want to try it. I've fixed and improved a few things, maybe several, depending on what version you are using.