Forget DOSBox, their programing is great, but their user experience is lackluster to say the least. Go for Boxer instead, it uses DOSBox for the emulation, but everything else is as Mac-like as it gets. http://boxerapp.com
Boxer has no built-in recording feature, but you don't need one. OS X comes with QuickTime Player, it can record what's going on on your screen. You can also trim the beginnign and end of the video and for more advanced editing go to iMovie (also included with every Mac). There are commercial screen recording solutions with built in video editors if you need something more professional, but but starters the QuickTime Player + iMovie combo should be more than enough.
To get a GoG game into Boxer first get the game files. You can either install the GoG installer on a Windows machine or ose Wineskin to create a wrapper and install it in there. Get the folder with the game files and drag&drop it onto Boxer's icon, Boxer will recognize the DOSBox setting from GoG and set everything up properly. You can then throw the Windows stuff (or the wrapper) away.