Look here for the basics of buying and selling.
Notice the TOTAL value of merchants inventory and gold is listed at the top above his inventory just under his picture. That is the value of everything he has and your side is the same - total value of everything you have.
Look in his inventory window and you will see his gold, the amount of gold he has in his possession is the maximum he has to buy goods from you. If you start selling him items the amount of gold he has decreases - eventually to zero. It will replenish after so many game hours / days (can't remember exactly right now). It will also increase if you buy from him since you are giving him gold for his inventory.
If he runs out of gold he will take what you are trying to sell him if you push the accept button. This is a feature which allows you to increase his opinion of you, give him stuff, he will like you better and give you better prices and pay more for your goods - until his Admiring is at 100% (in blue under his name when you hold the pointer over his character) After 100%, giving him stuff will not decrease prices you pay further or increase what he gives you for items further.
He also pays more for identified / repaired items than for unidentified / unrepaired items. He will not give you anything for some items (doesn't want them in his inventory / doesn't sell those items) and some items he will not buy or pay little or nothing for such as cursed items.
You should always press the scale button on both sides of the transaction before pressing the checkmark. If you don't have enough gold to buy something you can add other items from your inventory to the transaction (assuming you have items of value) until your side is equal to or above what he wants for the goods. If you are giving it to him just press your scale and the checkmark and agree to the dialog indicating an unequal transaction.
Not much of an explanation, sorry, it will become clearer as you play . . . =)
I will add to this as I remember other points. He will rarely pay you full value for an item since he is in business to make a profit.