This is by design. Per the helpfile:

RECALL prays for miraculous transportation from where you are back to the Temple of your hometown. If you have been fighting very recently and your hometown is very small, you may instead go to the nearest large city's temple.

You may not have realized your city is considered "very small".