I think I figured out what's going on here, here's what I think happened chronologically
-GOG finds the problem I mentioned in my edit above.
-to avoid this GOG rebind CapsLock to tilde and make their own "mapper.txt" that they install with the game
-GOG edit the manual to say "~" instead of CapsLock (a really unfortunate choice, I must say, this key is so differently placed on the various keyboards of the world)
-I once messed up editing the mapper.txt file and deleted it so that DOSBox restores the default, that's why run mode is now CapsLock for me and ~ for you.

Summary: What you need to do is bind R to CapsLock and it should work.

Edit: Oh, you already got it. Good night, then. :)

Bump.

shameless bump :)

Very helpful, thanks!

What a mess! ;)

I have a question. When I looked up this game on youtube it looked like you could select the level of difficulty for both puzzles and fighting. Is there no way to do this on the gog version?

There is. I think you just have to install the US version patch. There's a thread here about it: http://www.gog.com/forum/realms_of_the_haunting/transformation_patch_uk_us_version/page1

Then you get a prompt for both combat and puzzle difficulty when you start a new game. Combat difficulty governs enemy strength and health, while easier puzzles means the appropriate item will be automatically equipped when you click on the appropriate place in the level. This is handy for keys and obscure gems; you don't have to "try everything on everything" and is quicker if enemies are at your heels.

This post was extremely helpful. Thank you, Lafazar.

Very helpful post, thanks for the heads-up

If someone wants to use software scaling while fullscreen at the same time, here goes:

Execute "C:\games\Realms of the Haunting\Launch Graphic Mode Setup"
Choose "OpenGL", "Full Screen", "Keep Aspect Ratio" off, "Double Buffering" on, "Scaling Engine" to "hq3x", "Window Resolution to your native resolution.
In "C:\games\Realms of the Haunting\dosboxROTH.conf", set fullresolution=1600x1200 (or whatever native resolution you have).
Now if you switch from windowed mode to fullscreen and back (Alt-Enter), the fullscreen still uses the software scaler.

Any tips on Arcade Difficulty settings (in patched US version)? Should i just stick with Easy and enjoy the adventure without annoyingly hard enemies?