My guess is that GOG packaged the game files up after testing and accidentally carried over some save data. Anyone work out which file I need to delete?

Bump.

Files are in "$HOME/PierSolar" and in the .exe directory's game folder. It appears the game copies the files to & from those folders when run/closed. Unless the old .deb installer had a leftover it's probably from a previous install of yours (tar.gz & .sh are both clean).
Anyway, the file is likely PSHD.cfg as the others are for the saves. You would want to delete it from both folders. Also, I didn't actually test this.

I've never played the game before, so it's unlikely they are left over from anything on my computer.
Pardon me, but where would $HOME be on Windows? I've looked in the User directory and there's no PierSolar directory in there.

EDIT: Found it in C:\Users\<user>\AppData\Roaming\WaterMelon\PierSolar in the end, but no dice. Erasing it and all saves had no effect.

Game was released here for Win, Mac & Linux. I'm using the Linux build so I assumed you were as well. Then again the Linux build doesn't seem to have this problem.
Yeah, that AppData folder is the $HOME equivalent for Windows. You need to delete the file from both the AppData folder and the second copy of it in the install folder. The game does some sort of sync between the folders (under Linux anyway). Check the two folders again and delete the files from both of them.
The Mac build does have PSHD.cfg included in it's installer. I don't have the Windows installer in front of me else I would have just checked it. If also erasing it from the install directory doesn't help then I can try to take a closer look at the Windows build on Turkey day.

Oh no, I'd already found and deleted the one in the install directory, I just forgot to mention it in that post as it was much easier to find than the AppData one. No, deleting both didn't achieve anything so the achievements must be stored elsewhere...

Tried out the Windows build. No achievements activated. Forgot to check if the mini games were locked on not. Some achievements are visible (can see the requirements) and not hidden behind locked. How far have you played? Winning your first battle one you can get quickly if you want one to check.

Aaaargh, I'm such an idiot. That's what it was. I was seeing the named ones as being unlocked (since they contrasted with the locked ones). Sorry for wasting your time and thank you for helping me realise how dense I was being!

Heh, I probably should have suggested that from the start. No worries though. I stopped about halfway through on my initial play-through as I focused on another game I was playing at the time. I think now I will pick it back up again.