I've enjoyed the game so far, I've reached Gaia Garden and am finishing the dungeons on the top of the tree. But boy the difficulty has gotten very annoying at this point. I'm currently at level 47, and many enemies are below my level by a large margin, but those enemies still do a lot of damage and are giving me a really hard time. AND when I defeat those enemies I get like +1 experience point, which makes it very hard to level up.

I hate it that the game I've been loving so much is making me so angry. I'm considering lowering the difficulty, but that doesn't change the fact that this portion of the feels quite off.

I remember having some problems in Gaia’s Garden until I got used to the enemies patterns but nothing as serious as you are saying.
I’m not sure but for what you are saying maybe your equipment is not as good as should be?

In case you don’t know, there is handy feature in the equipment screen that allow to compare the overall level of your equipment with your current character level to check if it’s okay (I marked it with green in the attached screenshot), if you can the numbers should be green (the help screen of that equipment section explains how it works in detail)

I have 42 equipment vs 47 character, so I guess I have to further upgrade my stuff. But I'm already using Titan equipment, the best you can buy on the shop at Basin Keep.

There are some traders over there that have lvl 47 equipment, check if you have the materials that they want.

Glad it worked!

It's also interesting to check the buffs that the food could give to you, I found them very useful in the final part of the game.

If I may post it here, I've found a very easy (cheezy?) way to grind levels (for dat level 99 ballness):

Warp to Hermit's House (Bergen Trail), and go right to Rising Path 3. Start killing the 3 White Frobbits, and continue downwards to Rising Path 2.

Remember to kill the 2 Young Frobbits at the bottom of Rising Path 2 before back up and right to Old Plateau. Clear it out and continue upwards and back up to Rising Path 3. Stay on Rising Path 3, and do a lazy semi-cirle until you get back to the first 3 White Frobbits.

This wil net exactly 50 XP each round completed (plus gold earned). Keeping the S-rank means enemies will respawn by the time you complete the circle, so it's neverending.

1 round takes 2-3 minutes to complete. My personal best is +2150 XP this way (before I got bored) without the XP-counter resetting.

Bonus tip: play in Windowed-mode, put a long Youtube-video or other video up on another monitor (if available) and laze through the grinding.