I thought I had that problem too.

But then I found out that you could only change the resolution from the main menu's option screen.
If you try from within a running game, then it's grayed out, as you say.

So just do an auto-save, then press ESC and quit to the main menu.
There, change the resolution, and afterwards load from your auto-save, and enjoy the higher resolution.
Or change it when booting Nox, before loading your savegame.


ps: While some people seem to prefer the 800x600 over the 1024x768, I like the higher one better.
Mainly because the inventory, map, etc. don't cover as much of the screen, and because you can see farther.
Besides, my monitor enlarges an resolution to fit the whole screen.