You are not the first to try getting anti-aliasing to work. The problem is the way Longest Journey renders the 3D on top of the 2D which seems to be done by boxes surrounding the characters. AA applied to these elements results in these geometry problems and so far no one has found a solution.
You'd need to find a way to apply the post processing to the complete image. Basically you'd have to run the game in an emulator and after that use an FXAA shader. A way to do this would be to get Windows98 running in Dosbox (it's possible, but not easily) and then use ykhwong's build of Dosbox to apply the shader via the Direct3D renderer.
If you found another way - please share :)
Both nVidia's FXAA & AMD's Morphological filtering use post-processing filters to apply a slight blur to every single pixel onscreen in a D3d application when they are selected in the respective driver control panels. The effects work well in some games that cannot be AA'ed by traditional means, or older games just like this one. Unfortunately, this effect in this game blurs the text so much that it is barely legible.
However, in this game FXAA or Morphological filtering will work fairly well if you de-select "3d accelerator" at the beginning of the game when the options panel is presented. Morphological filtering (what I have) and FXAA will work fine on a game rendering in D3d software mode--which is what happens here when "3d accelerator" is de-selected and why it works. (Many people forget that D3d has a software mode, probably because it isn't used much these days.) The effect leaves something to be desired, still, but I think it's much better than turning on 3d acceleration and trying the FSAA the game because of the buffer/frame jumbling that occurs. It's easy to see the difference, at least with AMD's Morphological filtering, between D3d software mode with and without MF turned on. (I can't speak to how well nVIdia's drivers FXAA D3d software mode because I don't have a nVidia card.) I would be surprised to learn it doesn't work for nVidia owners, though.
Morphological filtering works fine for D3d software mode whether W-buffer or Force single buffer settings are on or off--makes no apparent difference that I can see. Text is still slightly blurry, but it's definitely legible--unlike when 3d acceleration is turned on in game and Morphological filtering is selected--wherein text becomes fairly illegible along with other undesirable graphics artifacts.
Best for me:
1) 3d acceleration turned off (which runs the game's D3d software mode)
2) Morphological filtering ON (FXAA for nVidia) in the the driver control panel
3) All other FSAA driver control panel effects turned off
4) If I set Game.exe to "Windows Service Pack 2" compatibility mode or I leave it unchecked so that it runs in the normal Win8.1x64 mode (no compatibility mode selected) the game runs and looks exactly the same for me
5) I set "disable high dpi" compatibility mode
6) I set "run as Admin" compatibility mode
The only real penalty to the D3d software mode is the shadow she casts is pixellated. But, compared to the jaggy-fest she is in D3d software mode without Morphological post-processing filtering selected, or in D3d hardware mode with FSAA & Morphological turned off--the grainy shadows are easy to put up with...;) It's a trade off, but it's an old game.