It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussionGOG standalone installer is single threaded and so slow(30 posts)(30 posts)
(30 posts)
Pages:
1
2
This is my favourite topic
Posted February 05, 2021
avatar
dtgreene: "Acceptable" quality doesn't need that much space
avatar
eric5h5: It does, though. We're well into the age of 4K resolution being common/standard. You can't get away with producing TV shows for 1080p anymore either.

In particular, degrading the quality significantly will not reduce it below acceptable levels.
avatar
eric5h5: It objectively does. Theoretically games could be distributed with smaller assets for people with old/bad hardware, but that becomes logistically annoying very quickly and generally not worth it.
No, quality levels of 1080p or lower are definitely acceptable.

(I would argue that quality only becomes unacceptable when it interferes with playing the game (for example, important things are hard to distinguish from background, or text gets hard to read), or if it's bad enough to cause discomfort in the player.)

I could point out that the company 6502 Workshop did just get away with releasing a game made for the Apple 2, with Apple 2 level graphics. And there's other games, like Shovel Knight, that are released with NES-tier graphics (adjusted for widescreen).

avatar
Z8X4T3JBMNV: I used the innoextract tool to investigate. There were 47.2GB worth of non-English language files not installed.
avatar
rtcvb32: It may be easier to pack a single installer with multiple languages and settings, vs individual languages.

I do remember there being individual language installs before, instead of your 100Gb installer you'd get 10 50Gb installers. On their end and those who enjoy downloading all the languages available it is quite a waste in contrast...

Would it be better or worse? I don't know.

Though i did come across a game with two game folders, one in Japanese and one in English. I used my new fancy md5 file comparison checker and it reduced the 400Mb Japanese game to 2.6Mb (A bunch of Json files, so all text, no audio), making note all the identical files from the English one. Allowing me to have both without the space wasted...

Each has their advantage and disadvantages... In the end there's no way to make everyone happy.
I think the best way to do it is as follows:
* For the Galaxy version, on install the client would present a list of checkboxes for languages and high quality textures (by default, the OS's language would be selected and no others, but the user would be able to change that); only those selected would be downloaded. (If a user chooses not to download a language, there would be no voices, but the text (including subtitles, which would be enabled by default) would still be present, as text does not use much space.) (Note that this would require the client to support adding the individual language/texture packs to an already installed game.) (Also, note that downloading the offline installer would be handled as below.)
* For the offline installer, it would be handled in a manner similar to DLC. You have the base game, the high quality texture pack, and language packs for each supported language (that has voices), and the user would choose what to install. Note that, while this would be handled like DLC installers, it would all be included with the base game.
Post edited February 05, 2021 by dtgreene
Posted February 05, 2021
avatar
dtgreene: No, quality levels of 1080p or lower are definitely acceptable.
To you, sure, but not the market at large. That's just not how things are any more; technology has moved on and we're in the 2020s now, not the 2010s. Besides, hard drives are commonly 10+ TB now, so space isn't really an issue.
I could point out that the company 6502 Workshop did just get away with releasing a game made for the Apple 2, with Apple 2 level graphics. And there's other games, like Shovel Knight, that are released with NES-tier graphics (adjusted for widescreen).
Yes? There's room for all kinds of games, and while I enjoy some retro-style titles, I'm not going to pretend they're the sort of thing that drives gaming.

However, none of this really has anything to do with GOG installers anymore so I'll stop now. ;)
Posted February 05, 2021
avatar
Niggles: Did they change the way the installers are compressed a way back?. At times small game installers sometimes take longer than the larger ones for me (even taking my system into account)
Yeah, there's been a fair clip of steadily pointless installer changes. Never once has GOG directly announced, "We've upgraded our MojoSetup to version X!"
Posted February 05, 2021
avatar
mqstout: There's only so much multithreading you can do for a disk-IO-bottlenecked activity like installing/patching.
Fair enough, but decompression can be multithreaded... and it's not.
Posted February 05, 2021
Honestly there is just no excuse for not having separate language installs for a game of that size, especially when they add up to so much wasted space and time.

It's worth boycotting the game over something like that. Just no respect for the end user.

What about all that wasted space when backing up the installer, and many of us wisely do so to more than one external drive. A criminal waste of space ... just no respect.
Post edited February 05, 2021 by Timboli
Posted February 05, 2021
To aggravate this even more, the installers are really flawed, let me explain:

I have my collection on my NAS.
I have a PC that has 2 drives. C: (Small SSD) D: (Big HDD).
When installing any game on the second drive (D:) the GOG installer (for any game) 1st unpacks all data on a temp folder on C: (wasting SSD write cycles) and then MOVES the data to whatever folder you chose (in my case, D:\GOG). So it also wastes time.

The temp folder it creates should be on the drive chosen, not on C: !!!!!!!!!!!!!

@gog please help!
Posted February 05, 2021
avatar
eric5h5: To you, sure, but not the market at large. That's just not how things are any more; technology has moved on and we're in the 2020s now, not the 2010s. Besides, hard drives are commonly 10+ TB now, so space isn't really an issue. Yes? There's room for all kinds of games, and while I enjoy some retro-style titles, I'm not going to pretend they're the sort of thing that drives gaming.

However, none of this really has anything to do with GOG installers anymore so I'll stop now. ;)
Commonly at 10+TB? Sorry, this isn't true. As someone who was recently in the market for computer parts (I just got my new rig put together at the end of last year) 2-4TB seems to be the norm. Beyond that is prohibitively expensive unless you have some REAL money to spend.
avatar
matcarfer: To aggravate this even more, the installers are really flawed, let me explain:

I have my collection on my NAS.
I have a PC that has 2 drives. C: (Small SSD) D: (Big HDD).
When installing any game on the second drive (D:) the GOG installer (for any game) 1st unpacks all data on a temp folder on C: (wasting SSD write cycles) and then MOVES the data to whatever folder you chose (in my case, D:\GOG). So it also wastes time.

The temp folder it creates should be on the drive chosen, not on C: !!!!!!!!!!!!!

@gog please help!
If you know the "temp folder" it uses, you can symlink it to a folder on your bigger HDD. If it is just your windows temp folder, you can move that anywhere you like.
Post edited February 05, 2021 by paladin181
Posted February 05, 2021
avatar
paladin181: Commonly at 10+TB? Sorry, this isn't true. As someone who was recently in the market for computer parts (I just got my new rig put together at the end of last year) 2-4TB seems to be the norm. Beyond that is prohibitively expensive unless you have some REAL money to spend.
It's very true, or else I would not have said it. The first result of a search for "hard drive" on Newegg is "Seagate Exos X10 10TB" for $205. The second is "WD Red Plus 6TB" for $156, and the third is "HGST WD 12TB" for $195. Anything below 4TB is rare, there's a bunch of 10-12TB drives, and a 16TB for $332. Hence, commonly 10+TB. Maybe you're talking about SSDs, in which case yeah, but you're not (hopefully) using SSDs as your storage/backup drives. I don't even have a 4TB drive filled and I have over 300 games, though granted the majority are not 50+GB games.
Posted February 06, 2021
avatar
matcarfer: When installing any game on the second drive (D:) the GOG installer (for any game) 1st unpacks all data on a temp folder on C: (wasting SSD write cycles) and then MOVES the data to whatever folder you chose (in my case, D:\GOG). So it also wastes time.

The temp folder it creates should be on the drive chosen, not on C: !!!!!!!!!!!!!
Indeed it should, but you can help yourself as paladin181 has suggested.

In fact, if you don't want to relocate your TEMP folder location permanently, you could do it temporarily using a script, that you run before and after your GOG installer. That said, probably no reason not to make a permanent change to another drive, unless that is an external one.

If you don't know how to do that, someone here might whip one up for you ... probably in a BAT file. I might even look into it sometime soon myself, as I am sure I'd want to use it too ... not installed any big GOG games yet, though I have plenty. That said, my Gaming Laptop has two internal drives, SSD and bigger HDD, so I would just change TEMP permanently.
Posted February 06, 2021
avatar
eric5h5: The first result of a search for "hard drive" on Newegg is "Seagate Exos X10 10TB" for $205.
dude, not everyone lives in the US where hardware prices are very, very competitive, in Europe and rest of the world, it's much more expensive to buy storage. Personally, I did jump on a few hdd deals in the US and even after paying for international shipping it came out cheaper than buying the same locally. Much cheaper. Doesn't mean everyone else would risk the same though, especially in these covid times.
Post edited February 06, 2021 by anzial
Posted April 11, 2021
I think this is related; I recently burned myself a nice copy of Torchlight onto DVD. When I went to install it from DVD, it was unresponsive for so many minutes I thought my drive was malfunctioning. Finally I sat there after double-clicking the install executable for about five minutes, and then it installed in two seconds. It's like it is silently unpacking with no indication that it's working, then suddenly just copying the install into place. Or did it load the entire game install into RAM and then just put it on my games drive once I hit install?
Posted August 18, 2021
Bump, little annoying with nvme, and even more, gog installer already use LZMA2 (in case of Horizon Zero Dawn installer)

OutputBaseFilename=setup_horizon_zero_dawntm_complete_edition_6278995_(64bit)_(44600)
Compression=lzma2

Just please check default ISS config [Setup]: CompressionThreads to auto, and don't use lzma2/fast option, because

A value of auto (the default) enables the multi-threaded match finder for all compression levels except fast, which doesn't support it.
Post edited August 18, 2021 by ilyvasile
Posted August 18, 2021
avatar
Z8X4T3JBMNV: There were 47.2GB worth of non-English language files not installed.

With this amount of bloat, surely it would have been smarter to make installers for each language.
Yes it would, but the trade off would likely be the increased support requests from people who downloaded the wrong language. At least the excess data isn't included in the install itself (I recall Supreme Commander which included tutorial videos for each language - deleting the unneeded ones saved me over 2GB).
avatar
Z8X4T3JBMNV: I also note that it is a bit strange for a Game to be using compressed files in the final product, especially ones so heavily compressed, because this adds overhead to the game performance as the computer (or console) has to spend CPU time in decompressing the files rather than putting that CPU time towards running the game itself.
Games have been using compressed data for decades - Doom with its .WAD ("Where's All the Data") format for example. Aside from speeding up data retrieval on slow disks, it reduces file system overheads by having a couple of files rather than hundreds of thousands for each individual resource.
avatar
Z8X4T3JBMNV: But that's not all, looking at the Task Manager while it is verifying or installing, it is single threaded. I have a CPU with 12 Threads so I got 11 Threads here doing nothing. It should be 12X faster than it is if it were using a modern compression algorithm.
Installers generally are limited by media read/write speeds. So multithreading wouldn't make a jot of difference in most cases.
avatar
Z8X4T3JBMNV: These GOG installers are straight out of the 00s and just don't scale with modern games which are dozens of Gigabytes big.
The only difference an updated installer would make is throwing more adverts in your space - the very first GOG installers used RAR compression which is pretty good for most cases, and required less overheads for their advertising.
avatar
Z8X4T3JBMNV: It doesn't help in this case that whoever packaged Cyberpunk in particular didn't pay much attention to what they were doing to realise that the compression gains were negligible and should have turned off the compression if possible, let alone including all the language files in the main installer when each one is about 4.5-5GB (and there are 11 of them).
Cyberpunk seems to be very much a "work in progress" and the "final build" (should that ever come) will probably have more attention paid on the installer side.
avatar
dtgreene: At least they're not 16-bit installers, which won't run natively on 64-bit Windows...
Oh yes they will.
avatar
ilyvasile: Just please check default ISS config [Setup]: CompressionThreads to auto, and don't use lzma2/fast option...
You do realise that the better way to raise this is via a support request which is more likely to be read by GOG staff, than a forum post? (which, unless reported to the mods for abuse, almost certainly won't).
Posted August 18, 2021
avatar
AstralWanderer: Installers generally are limited by media read/write speeds. So multithreading wouldn't make a jot of difference in most cases.
Although media read speed does play a part, GOG's installers still do install noticeably slower (up to 4-6x) than unzipping a zip file of the game folder. Some of us tested this stuff in the past when GOG changed their internal InnoSetup format (instead of storing them as files, they store them as "Galaxy streams" that have to be unpacked, then reassembled and then installed):-
https://www.gog.com/forum/general/dosbox_linux_support_dropped/post27
https://www.gog.com/forum/general/offline_game_installers_why_use_ctemp/post17
Posted November 27, 2021
avatar
Z8X4T3JBMNV: I don't know if Galaxy is any better but using the standalone version of Cyberpunk, and it takes hours for me to download it (not your fault), about an hour to verify it (because who knows after downloading something for days), and about an hour to install it.

You'd think that it is using some crazy compression, but nope.

The sum of all the install files: 103GB

The sum of all the files after install: 59.6GB

That's right... the installer is bigger than the final product.

I used the innoextract tool to investigate. There were 47.2GB worth of non-English language files not installed.

With this amount of bloat, surely it would have been smarter to make installers for each language.

And still 59.6GB+47.2GB=106.8GB, so all this long compression times just to save 3GB. You would save space just by splitting the installers by language. Save GOG some bandwidth costs as well.

I can only deduce that the .archive files in Cyberpunk are already heavily compressed, at least by the standards of whatever compression algorithm the GOG installer is using, it can barely compress anything out of it if anything at all (maybe the compression gains were from the non .archive files)

Specific to Cyberpunk, I also note that it is a bit strange for a Game to be using compressed files in the final product, especially ones so heavily compressed, because this adds overhead to the game performance as the computer (or console) has to spend CPU time in decompressing the files rather than putting that CPU time towards running the game itself. Maybe the Cyberpunk Developers could get more performance by not using such strong compression on the Game Files.

But that's not all, looking at the Task Manager while it is verifying or installing, it is single threaded. I have a CPU with 12 Threads so I got 11 Threads here doing nothing. It should be 12X faster than it is if it were using a modern compression algorithm.

These GOG installers are straight out of the 00s and just don't scale with modern games which are dozens of Gigabytes big.

It doesn't help in this case that whoever packaged Cyberpunk in particular didn't pay much attention to what they were doing to realise that the compression gains were negligible and should have turned off the compression if possible, let alone including all the language files in the main installer when each one is about 4.5-5GB (and there are 11 of them).

To be clear, the once the game is installed, the other 10 languages are not even copied over at all, the the game runs perfectly fine without them.

With all this extra time downloading unnecessary language files, verification of that big download (optional), and installing it (where the installer is decompressing everything on a single thread, on files which are practically already fully compressed in the first place and will stay fully compressed) this all adds up to a huge waste of time of many hours multiplied by however many thousands of people used the GOG standalone installer, and who waited for the Cyberpunk download to complete with all these extra unnecessary files (languages they don't need). I wonder what that would add up to, it would be quite a lot.

If nothing changes, it is just going to happen again the next time there is a big game.

I kindly suggest updating the GOG installers to utilise lzma (7-zip/.7z) format for compression, with multiple threads, which actually a very easy task to do given the amount of resources available for that. And please ask the person packaging them up to kindly pay close attention to 1. The overall download size where possible to exclude files and 2. Not to use strenuous compression techniques on files which can hardly be compressed any further.

Thank you very much.
You are absolutely right; it's so slow that I am seriously question ing why I would keep giving GoG my money; because DRM is less of a hassle using Steam/Origin/Uplay for newer games and Ebay/dosbox for older games Compound this with what for mehas been the most awful customer support of any platform; it's looking like a walk away while ahead situation.
Pages:
1
2
This is my favourite topic
Community
General discussionGOG standalone installer is single threaded and so slow(30 posts)(30 posts)
(30 posts)