I can't tell if the game cares whether I steal from people's houses and sleep in their beds, for example. Or for killing non-evil creatures like sharks. If I find a scroll in someone's bookcase and take it, my "karma" doesn't change afterwards, so does that mean there is no penalty?

I assume killing people is bad but I'm wondering more about edge cases - the manual says that sleeping in someone else's bed without permission is a possibility, but I don't know what the consequences are if they catch me and kick me out later.

I'm not sure if this is a spoiler question or not, but this is my first time playing U5 so if you can answer without giving anything away that would be nice. I just finished my 3rd playthrough of U4 and I'm trying to figure out what has changed with the virtue system between the two games.

There is stuff you can get away with taking, but the game does punish you for taking some things. Minor loot items may not be an issue, but things like looting a treasure chamber certainly do incur penalties. Note that you will find quest items by searching homes and businesses, and those are fine to take.
One that seems illogical is that you are punished for taking food out of the fields even at Iolo's hut at the beginning of the game, with the owner in your party.

Taking stuff that someone owns is stealing. If no one owns it, then you can take it without penalty.

I'd imagine things like sleeping in someone's bed is similarly a problem, but sleeping in an 'unclaimed' bed isn't an issue.

I haven't actually played this game, so some of this is guesswork on my part.

At the start of the game, you have 75 karma. Certain virtuous actions will increase it, while certain non-virtuous actions will decrease it. In particular:
* Taking a treasure chest in town will typically cost you karma. The chests in the castle's basement will, I believe, cost you 2 karma each, but it's usually worth it.
* Voluntarily giving money to a begger will give you karma, though you need to leave and re-enter town each time.
* Donating money at a shrine is a quick way to get karma, but it is expensive (so only makes sense if you have extra money).
* Running away does *not* cost you karma, and there is no need to worry about accidentally killing non-evil enemies; those aspects of Ultima 4's virtue system were not carried into Ultima 5.

Now, as for effects of karma:
* If a character dies and is revived, that character will lose experience based off your karma; at 0 karma, the character's XP goes down to 0, while at 99 karma, no XP is lost. The character's level will be re-calculated, but the character won't actually lose (or gain, if this results in a level increase) any stats (other than HP) as a result. This can result in a level increase. On the other hand, you can abuse this behavior to max your stats; gain a level normally, then lose it through death, provided that your karma isn't 99.
* If your entire party wipes, everyone will be revived with the appropriate XP loss (and level recalculation), and then your karma will be set to 75 if it's lower.

Ultima 6's karma works similarly, but there's no level recalculation and there don't seem to be repeatable ways to gain karma the way there are in Ultima 5.

Edit: One more thing: I believe that your interactions with the guards won't cause karma loss regardless of whether you pay up, fight, or go to jail.