I'll probably regret this, but: the chaos spawns aren't aggro if you flee from them quick enough (they don't attack when you return to them).
.