I don't really understand how the weapon stats work.
When playing HotU I was playing a cleric and was using the Planar Mace, which I has Enchantment Bonus vs.: Outsider +4 and I had upgraded it with Haste, Keen, 1d6 Fire damage, and Enhancement Bonus +6.
However I got to a point where a strong monster was blocking my path and no matter how I tried I just could not beat it and would rarely even hit it.
I then read on a forum that if I used the Astral Blade +8 (Damage Bonus: Sonic 2d6 Damage, Enhancement Bonus: +8) which I had sold to the gene merchant, the monster would go down easy. Lo and behold, I killed it on the first try.
The reason I sold the Astral Blade in the first place was because even tho it had a higher enhancement bonus my Planar Mace had other bonuses that I taught would make it a better weapon (and I had been using it for a long time so I had become attached).
So where did I go wrong in my thinking? Is enhancement bonus the thing you should look at first when choosing a weapon? Or is it some other stat?

Enhancement is very important because it decides how good is your your bonus to hit and damage rolls. With enhancement +8 you have +8 to your attack rolls and +8 to damage rolls. But again +6 is still a strong enhancement. Maybe you were just extremely unlucky on your hit rolls? +2 difference isn't that much unless the monster had some heavy resistance which required +7 and above enhancement to beat him. Are you talking about the final boss in HotU maybe?

No, it was the Guardian of the Path in Puzzlers Sepulcher, and the forum post says that you need a +7 to beat him.
But when choosing between a +6 weapon and a +8 is it better to look a enhancement bonuses first and then at the other bonuses?

Not quite. Most of the enemies don't have major resistances, so if the other bonuses are very good it might be a better idea to use the weapon with them even if it has lower enhancement bonus. Just be sure to keep something with bigger enhancement just in case you run into enemies that have a high resistance. Your Planar Mace +6 is better than Astral Blade +8 against most enemies.

The monster probably had Damage Reduction. Check out the article in the NWN Wiki, it will shed some light on how it works and why the enhancement factor of your weapon can be crucial sometimes to beat an opponent with Damage Reduction. You can also equip items with Damage Reduction yourself, if you find some, giving your character the same resistance towards low enhancements on your opponents' weapons.