Assuming you're using the unmodded base game, the below should hold true. Mods can (and do) change some of these. I'll provide what I think are the values when I know them, but I might be wrong (I haven't played in years, and I haven't played unmodded in longer).

First things first: Relations are complicated. There's a bunch of different factors that may play into whether a given stack or town is happy, content, stable, or ready to revolt. Also, relations will move 1 point toward their base value each turn.

Be aware that race relations are NOT symmetrical. The Elves might like the Human leader, but Humans might not like the Elf leader.

Also note that terrain may have an effect. I don't recall how much of this was active in the base game, but some of the mods make it a much bigger deal to have units or towns on terrain they don't like (ex. Frostlings in desert).

-Migrating a town TO a race will raise relations with that race by +10. This only takes effect on the turn in which the import is complete.
-Migrating a town will reduce the relations with the prior race that matches that town by -15. This takes effect on the turn in which the exporting begins. Cancelling the migration will improve relations by +15 (effectively negating the minus).
For example, migrating an Undead town to Elves will make the Undead relation drop 15 points and the Elf relation rise 10 points. If the Elves start migrating an Undead town to Dwarf, but the Humans capture the town before the migration is complete, then the Humans can either cancel the migration to gain +15 points with Undead or they can allow the migration to complete for +10 points with Dwarves. Either way, the Elves retain the -15 relations with Undead.

-Looting or Razing a town will harm your relations with that town's race by 30 points. This is a big change, so don't do this to a race you want to keep friendly (unless you have a huge buffer, I guess, but then why are you razing it?).
Ex. Loot/Raze an Undead town and the Undead will move quite a bit closer to Wary or Hostile.

-Building or upgrading a wall around a town will improve relations with that town's race by 5 points. Be advised that the credit goes to whomever owns that town when the wall is completed!
Ex. Build a wooden wall around an Elf town for +5 relation points (and a wall). Upgrade the wooden wall to a stone wall for an additional 5 relation points. If the Undead start building a wall around an Orc town, but the Humans capture the city before the wall is complete (and they don't cancel the construction), then the Humans will enjoy +5 relations with the Orcs.

Here's the biggest "It's Complicated" in race relations:
-When treaty relations between two leaders change between Alliance, Peace, and War, then race relations will be impacted. Making Peace or an Alliance will improve relations, while breaking an Alliance or declaring War will damage relations. However, relationship changes don't just happen between the two affected races; other races that really like or hate one of the treaty participants may experience a relation change as well. Races that feel the same way about both treaty participants probably won't experience much or any change. I don't know all the math, so it is easiest to just look at the values each race will experience when you're considering the diplomatic action. In general, the two races involved will have potentially impactful relation changes (depending on how much they already liked or disliked each other), but the relationship change with anyone else shouldn't be very impactful unless it was already on the cusp of changing a relation status (Friendly, Polite, Neutral, Wary, Hostile) (different statuses have different possible effects related to either acquisition or retention of troops and cities).

Ex. Assume default relations for this. If the Elf leader makes a peace treaty with the Orc leader, then Orcs will like the Elf leader more, and Elves will like the Orc leader more. To a lesser extent, Dwarves, Halflings, and High Men will dislike the Elf leader for playing nice with someone they don't like, while the Orc leader may experience a dip in relations with Goblins, Dark Elves, and Undead. The Neutral races probably won't care about the treaty and experience no change with either side. Conversely, Goblins, Dark Elves, and Undead may like the Elf leader a little more, while Dwarves, Halflings, and High Men may like the Orc leader a little more.

These race relationship changes only happen when a diplomatic is formalized. This means that both sides have to agree to a peace treaty or alliance, but an opponent could potentially jeopardize your race relation by breaking an alliance or declaring war on you! This isn't always relevant, but I vividly recall one game where I was on the cusp and had to hope another player didn't declare war and screw my relationship with their race before I could improve it enough to keep the captured cities and recruited units.

-In general, it is easier to damage relations than to improve them. For example, migrating a city back and forth between two races will ultimately make both races hate you (the cost of being exported is greater than the benefit of being imported). Likewise, breaking an Alliance or declaring War damages relations more than making an Alliance or making Peace improve relations.

-The Tranquility spell will improve race relations between all races. I don't know the details beyond that it improves each turn by a few points. I don't know whether it is hard capped at some amount of improvement, and I don't know how fast relations return to normal if/when Tranquility is no longer in effect.

-Hatred does the opposite of Tranquility, reducing relations between every race. I used the Hatred spell in a game a some years ago; as I recall, there was a cap on the change, but it was quite large.

-Machines without a race don't care about relations.

-Units of type "Creature" only care about their alignment compared to your alignment.
Ex. It doesn't matter how much work the Undead do in order to achieve good relations with the Azracs, those Tier 3 Azrac creatures are never going to like the Undead leader. I'm not bitter, why do you ask? :(

-All those race relations do come into play when stacking units of different races together. Again, there's a whole complex math happening behind the scenes that I don't fully understand, but stacking units of races that hate each other may cause part or all of the stack to decide they'd rather freelance than work for you. Each unit's morale is affected by whom they are stacked with, and the starting morale depends on how they feel about you. The party morale is a combination of the accumulated individual morale, but each individual morale's effect is weighted based on how strong that unit is (or how strong that race's accumulated morale is?). A large group of weaker friendly units can keep a single stronger unit from rebelling, or a single/few strong unit(s) might keep a bunch of weaker units marching in line.

Similarly, I think cities may be affected by having guards of a race they don't like, but maybe I'm remembering that from a mod or a different game.

-----

Speaking specifically to your experience above, individual maps may have race relations set to some given value. Also, if a race loves or hates you enough, then all the above won't be enough to make much difference in a race working for you. You can check the various relations leaders have with races and that races have with leaders on the diplomacy screen by clicking that leader or race. You can mouse over a portrait to get a number value.

The devil is in the details. To give a more specific answer to your case, I would need more specifics on which maps, whether it was a campaign, what relations the leaders and races had, and what diplomatic actions were being taken.