On Windows 7 at least, the save files are stored in [user name] --> AppData --> Local --> VirtualStore --> Program Files --> GoG.com --> Ultima Trilogy (or whatever folder you installed the game to.) You'll have click the "Organize" tab and go to "folder and search options" and check "Show hidden files, folders, or drives" to view the AppData folder
In the Ultima 2 folder there's a file called PLAYER; if you move or delete it the game will make a new one when you create another character.
Honestly, it's not worth the trouble because Ultima 2 is a really, really terrible game. Which is too bad because I like the whole time travel thing.
You've got a good point; while I loved the game when it came out and spent most of my freshman year in High School playing through it, it has not aged well at all. Ironically when Ultima III came out (which is a much better game) I was disappointed because it was much too hard compared to II. II had very little grinding needed and the worst danger was running out of food since your character eats like a pig and starves as soon as there is no food.