The TBS genre grew immensley since 1996 and there are so many more complex and varied games than HOMM2 (I highly recommend Endless Legend), so forget about strategy - the main reason why H2 is such a great game is, that it has a soul. The story-book like graphics, the tiny details (elaborate portraits of each hero, the hand-drawn animation of each unit on the battlefield, etc.), the unique town themes, the amazing soundtrack - they all make it such a memorable game.
I've been playing it since I was a 12 year old kid in 1996, and it's still my favorite game ever.
-----------------
Technical comparison to H3 for veteran players:
Yes, Heroes 3 is bigger and has more of everything. But upon closer examination, there are things which H2 does better than H3:
Towns are less self-sufficient (lower income, lower marketplace ratios) - encouraging the player to explore more and to take chances. Low tier units are not as disposable as in H3, making you to appreciate them more instead of treating them like cannon fodder.
Ultimate artifacts effecting heroes instead of towns, again encouraging player to take action against stronger opponents instead of camping for longer periods of time.
Factions are less balanced - encouraging the player to take different approaches instead of sticking to the same game plan in each playthrough.
All in all, H2 is less formulaic, making it a better game in the long run imo. But all of that is just "nice to have" - again, as subjective as it sounds, H2 has a soul unlike any other.