It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
So here is a topic I often think about. Being a digital hoarder, and an organisation freak, I always reassess how I am organising my backups. So this is twofold, actual storage organisation and then tools to work with that.
Actual storage:
So, I currently have two sections, with the following folder structure:
Games
<first letter>
<series> - optional
<game name> [<year>] [platform]
<store>
Docs
Mods
Patches
Multimedia
Film
<film name> [<year>] [<tags>]
Music
<artist>
<album>
Tv shows
<show name>
Season <x>
Music videos
<name>

So this seems to work ok, am debating on wether or not to divide up movies etc. to use first letter like games. Anyone else have their own setup that they want to show? I find it is far easier to work with a fixed folder structure programmatically.


Tools:
For the last couple of years I have been using collectorz game library. It’s ok but it’s subscription based and closed. Have looked at a few things over the years but nothing seems to fit my requirements. Launchbox looks nice, but no import of metadata, also no direct access. Excel, it’s really not a database and using it as such will corrupt your data eventually.
So I think I will go with a two pronged approach. In the immediate future have sqlite studio as a front end for an SQLite database, perhaps build some tools to work with the folder structure (eg check what is in dB matches what is on disc, maybe md5 checks etc.). Then later on will build my own front end (when I have some free time) and incorporate the various tools into it.
What do others use?
For my purpose, I have a list of requirements which hasn’t helped:
Offline
Open backend
Ability to run tools, or script in the system
Not a proprietary client

Just interested what others do.
This honestly seems like too much effort for my tastes. If you're going that far, why not use something like KODI and let the software figure it out for you?

I have a folder on my external drive called "GOG Games". That's the extend of my organization.
avatar
nightcraw1er.488: I always reassess how I am organising my backups.
Organising backups? A backup is just a copy of your main storage.

btw. Launchbox now imports metadata from GOG.
avatar
Darvond: This honestly seems like too much effort for my tastes. If you're going that far, why not use something like KODI and let the software figure it out for you?

I have a folder on my external drive called "GOG Games". That's the extend of my organization.
Nice, had t heard of that. Will have to take a look. As I mentioned above I have tried out lots of these types of things over the years and they all seem to fail on some points, hence why I am likely to do my own thing, but will look at this first.
avatar
nightcraw1er.488: I always reassess how I am organising my backups.
avatar
teceem: Organising backups? A backup is just a copy of your main storage.

btw. Launchbox now imports metadata from GOG.
Well, a backup is a backup of anything. My main storage is a back of gog, humble, itch etc. Games, as well as various others things. Organisation is key to any endeavour, learnt that the hard way when I first organised what was just a disparate mess of folders.

Yes, I know launchbox can import gog metadata. Unfortunately (as mentioned) I use collectorz, which can export to a couple of formats, and has around 15k records currently (30+ years of hoarding!), so the gog part is relatively small. Need a good importer or access to the backend, and if I am not mistaken there is no direct dB access? Also, launchbox is more of a launcher, and that isn’t what I am after, just an organiser.


Back on the main items:
In terms of music (I am currently consolidating and updating this folder structure with the movies etc.) I have found this tool:
https://picard.musicbrainz.org/
Not sure if it’s any good, only just found it, but it can apparently rename and move files based on the provided metadata and tags, which could be useful, currently use musicbee (which is great and has helped me get a lot of the metadata in) and vlc (for conversion), so hoping that might be all I need there.

Also, I want to add some tools in, auto checking folder structure, md5 checks etc. so another reason I might need to do it myself. Might do it in go just to learn that, maybe with a c# front end and sqlite or xml backend. Now it’s just finding the time...
Post edited March 22, 2020 by nightcraw1er.488
avatar
nightcraw1er.488: Just interested what others do.
For games I basically backup : The game itself (plus .md5 of installer files), Soundtrack, Patches, Manual / Walkthrough (if available), Icon, Mods / Community Content, Source Ports, and if required a note of any applied tweaks (.ini edits, etc) stored as one game per folder under Games.

Exceptions are made for many smaller portable games that go together naturally as having one folder filled with different zips (or multiple games in one zip) makes it very easy to batch extract dozens / hundreds of related smaller games at once with a few clicks often with any custom settings / key-bindings intact without needing to manually install each one at a time. Eg, dozens of ScummVM games are inside one \ScummVM folder, Doom Engine games = Doom 1-2, Heretic, Hexen, Strife, Hedon WADS are stored alongside pre-configured GZDoom in one zip file with custom settings, key-bindings, Start Menu Links, etc, all done in one action. 100x DOS games = 6 clicks vs 600 clicks (and custom .map and .conf (DOSBox controller / keyboard rebindings & config files) are retained between installs. It saves so much time when reinstalling.

As for launchers / clients, I don't use any of them at all, just the Windows Start Menu Links (with Open Shell - formerly Classic Shell) with games typically arranged alphabetically within "folder genre's" (FPS, RPG, RTS, etc). I'm not bothered about artwork / fancy 10ft HTPC UI's, etc, and as a result it's faster to scan through lists of text names when I see dozens of them on-screen at once.

As for movies / TV, it really depends on how many you have and how much effort you're willing to put in adding tags / meta-data. I have a small collection of classics / favourites, but have never really been a Home Cinema enthusiast or movie-buff who own thousands of them so simply storing all files in one "Movies" folder and browsing the files in thumbnails view in Explorer had never been an issue for me without feeling the need for HTPC software.
I only backup what I really care about, mostly RPG and FPS classics, so alphabetical works fine for me. I have some point and click adventure backups too but most of them are in a big ScummVM folder.
avatar
nightcraw1er.488: Just interested what others do.
avatar
AB2012: For games I basically backup : The game itself (plus .md5 of installer files), Soundtrack, Patches, Manual / Walkthrough (if available), Icon, Mods / Community Content, Source Ports, and if required a note of any applied tweaks (.ini edits, etc) stored as one game per folder under Games.

Exceptions are made for many smaller portable games that go together naturally as having one folder filled with different zips (or multiple games in one zip) makes it very easy to batch extract dozens / hundreds of related smaller games at once with a few clicks often with any custom settings / key-bindings intact without needing to manually install each one at a time. Eg, dozens of ScummVM games are inside one \ScummVM folder, Doom Engine games = Doom 1-2, Heretic, Hexen, Strife, Hedon WADS are stored alongside pre-configured GZDoom in one zip file with custom settings, key-bindings, Start Menu Links, etc, all done in one action. 100x DOS games = 6 clicks vs 600 clicks (and custom .map and .conf (DOSBox controller / keyboard rebindings & config files) are retained between installs. It saves so much time when reinstalling.

As for launchers / clients, I don't use any of them at all, just the Windows Start Menu Links (with Open Shell - formerly Classic Shell) with games typically arranged alphabetically within "folder genre's" (FPS, RPG, RTS, etc). I'm not bothered about artwork / fancy 10ft HTPC UI's, etc, and as a result it's faster to scan through lists of text names when I see dozens of them on-screen at once.

As for movies / TV, it really depends on how many you have and how much effort you're willing to put in adding tags / meta-data. I have a small collection of classics / favourites, but have never really been a Home Cinema enthusiast or movie-buff who own thousands of them so simply storing all files in one "Movies" folder and browsing the files in thumbnails view in Explorer had never been an issue for me without feeling the need for HTPC software.
Yep, exactly what I do. Though I do need to go through the doom and elder scrolls folders and organise those a bit.

In terms of movies, yeah, not a lot, less than a hundred, that’s what I haven’t subdivided, but as with all these things it’s generally good to think of before it gets out of hand. Not too bothered about software, I only occasionally watch things, so pot player is fine for that purpose. Main lump is games and music.
avatar
StingingVelvet: I only backup what I really care about, mostly RPG and FPS classics, so alphabetical works fine for me. I have some point and click adventure backups too but most of them are in a big ScummVM folder.
Yep, I have a hoarding problem :o)
But it’s a bit more than that. For instance there are times where you want to keep a specific version, or items you may not get hold of again. I had a dvd full of Amiga roms, which seems to have disappeared over the years, can’t be bothered to go through and download them one at a time, hence why I have massive unorganised spectrum, amstrad, and c64 libraries, just old copies never really gone through but don’t want to lose them.
Just years of stuff, personal and clones, still got a load of compilation disc images from back in the days of magazines, FPS, adventure compilations etc. Will one day go through and remove what I have elsewhere.
Post edited March 23, 2020 by nightcraw1er.488
avatar
Darvond: I have a folder on my external drive called "GOG Games". That's the extend of my organization.
Same.

My physical games are a bit useless because most of the times I need to look online for patches and tutorials otherwise it's not working on my modern PC, but I only backup the games that I really love and "can't live without".

I may surprise a lot of people but I still have a DVD drive on my desktop PC that I often use to watch series or listen to my favorite music. The only backups that I have are the ones of my phone for when I'm on the go.
avatar
nightcraw1er.488: Yep, I have a hoarding problem :o)
Or you just have really wide interests. I'm pretty focused on just PC stuff and only a handful of genres. Other people play lots of everything and that's gotta be a bigger backup nightmare.

Another big reason I only have classics backed up (and a lot of mods/extras) is because of HDD space. One day soon I'll get a couple massive HDDs and back up way more stuff.
avatar
nightcraw1er.488: Yep, I have a hoarding problem :o)
avatar
StingingVelvet: Or you just have really wide interests. I'm pretty focused on just PC stuff and only a handful of genres. Other people play lots of everything and that's gotta be a bigger backup nightmare.

Another big reason I only have classics backed up (and a lot of mods/extras) is because of HDD space. One day soon I'll get a couple massive HDDs and back up way more stuff.
I have 3 of these:
https://www.amazon.co.uk/TerraMaster-D5-300-External-Enclosure-Diskless/dp/B01KO03BBA/ref=mp_s_a_1_5?keywords=terramaster+d5&amp;qid=1584950980&amp;sr=8-5
Each full with either wd red 6 or 8tb drives, plus 3 8tb seagates in the main machine. Periodically backup from main to the external, then moved from house. I use freefile sync. Can definitely recommend the terra master raids, great product. The main drives are merged into one big drive using windows combine drives, seems to work ok. Plus I have other external drives and things which are not part of the main backup process. It seems a bit overkill, but it’s actually real easy once you have it working and doesn’t take much time (though it’s been years getting to this stage).
avatar
nightcraw1er.488: Just interested what others do.
I call them "archives" and not "backups", meaning that is where I archive files which I want to keep for a longer time. With a backup I understand more like making (daily or weekly) backups of my daily files on my PC so that if the PC or its filesystem breaks up, I can get the files from the backups.

I try not to depend too much on third-party programs because I want the archives to be usable and accessible both e.g. on Windows and Linux. I try to keep my archives simple and not to use too much time trying to arrange them, mainly so that I can find something easily from here if needed, and throw new stuff I feel I want to keep into there. I see no reason trying to separate or arrange files alphabetically or anything; File Explorer arranges them alphabetically anyway, and if I know there is a certain file (game, movie, piece of music, whatever), I just need to enter it to the File Explorer search box and voila, it points me to where that file is. No further magic or extra applications needed.

My setup mostly is:

1. 2x 8TB HDDs (sitting on an USB3.0 HDD docking station, one at a time usually).
- Here goes most of the files I want to keep from over the years, mainly games (in digital format) and and "computer stuff". It simply has some subfolders like "PC_Games", "Console" (including emulators, MAME, Amiga etc.) etc.

2. 2x 3TB HDDs (sitting on that same USB3.0 HDD docking station)
- Videos/movies, music, personal photos etc.

3. A separate 5TB USB HDD for my GOG game installers.

So I have two identical HDDs which are basically 1:1 copies of each other, except for my GOG installers (because for those I have the second backups on GOG servers online :)). Every now and then, if and when new stuff has been added to those archives, I copy all files from the primary HDD to the secondary, and then the secondary becomes my new primary archive HDD. So nothing fancy, no RAID or mirroring or anything, just copying all the files all over again to the other HDD. This also refreshes the filesystem and removes any fragmentation. I guess I could use also rsync or similar to copy over only new and changed files, but then removing obsolete files becomes more complicated. I keep it simple, just copy over everything every now and then from the primary archive HDD to the secondary.

For data integrity:

- For the games and such, I use compressions (like rar or nowadays 7-zip) as much as I can so that I can always verify single files that they are still ok. For videos or music this is not very useful though (they don't compress well and they need to be easily accessible, without having to uncompress them before use).

- i use also dvdsig (which is Windows-only) and rhash (both Windows and Linux) to keep track of hash checksums of all the files on those archives. dvdsig is more for keeping a checksum of files within certain subfolders, while rhash is for making a full list of checksums for the whole archive.

- Currently those archives use NTFS filesystem, but my plan is to move to using either OpenZFS or BtrFS filesystem at some point, as they have their own data integrity checking. That way I wouldn't have to manually use dvdsig or rhash anymore, but the filesystem itself would keep track of each file's checksum and that all the files are intact. Easier to track that the archives are still ok.
Post edited March 23, 2020 by timppu
avatar
timppu: ...snip
Interested use of the hash. The rest of your setup sounds like what I had some 6 years or so back. I started using freefilesync as the updating of each clone became a real nightmare, so I either copied all 10tb each time or things got messed up. Now I only update what has changed which is much quicker.

The other thing I like about the raid boxes is that it’s easy to unplug it and remove from the premesis and store. I have some plastic hard case for hdds, but having to undock and package each time becomes a pain.
avatar
nightcraw1er.488: Interested use of the hash. The rest of your setup sounds like what I had some 6 years or so back. I started using freefilesync as the updating of each clone became a real nightmare, so I either copied all 10tb each time or things got messed up. Now I only update what has changed which is much quicker.

The other thing I like about the raid boxes is that it’s easy to unplug it and remove from the premesis and store. I have some plastic hard case for hdds, but having to undock and package each time becomes a pain.
Yeah I guess it depends on the scope. I've tried to cope with as simple and cheap setup that I can, but maybe at some point I will upgrade to something more (RAID, NAS etc...).

Currently my biggest interest is to move to some more advanced filesystem that is "corruption resistant" so that I wouldn't have to keep using utilities like rhash and dvdsig manually, but I'd need to try them out properly on my own, also whether those filesystems can be at least read on Windows.
I use gogrepo, and sometimes run verify. Unfortunately GOG provides md5 only for the installers, not for the extras.
avatar
blotunga: I use gogrepo, and sometimes run verify. Unfortunately GOG provides md5 only for the installers, not for the extras.
I think in the case of zip files, gogrepo checks that the zip file test is ok. I guess that is good enough confirmation for zip files (IIRC zip uses internally CRC32 checksums, not MD5).

The only thing then is any extras files which are not zip-files, I guess.