Two out of my three villagers didn't ever go to Arboria, or similar places such as Trothon because of this fact.

It all depends on the villagers' RP. Hardline, militant villagers probably shouldn't be using magical portals of any sort.