How important is Memory compared to pure Intelligence? I've played the first chapter as a pure fighter type and wasn't interested in that playstyle, so I decided to switch to magic and see how much that interests me.
Well, you need enough memory to have 2 or 3 turns worth of spells available since you won't be using autoattacks after the start. Add a few buff and mobility spells and you'll want to have at least 20 memory slots. At level 20, you'll have 13 base slots so getting to 20 memory will give you 23 slots total. That should be enough.
Intelligence should be maxed ASAP for pure damage. So the progression is to get only enough memory to get the spells you must have and the rest into INT. After that add wits for initiative and crit chance using the savage sortiledge talent.