If the background checks of the village are so poor that I and a mage travelled with you one day, and the next time I'm on (which was a couple of days later) you are a rager, it suggests that they did not realise you were in the habit of travelling with mages so recently.

Perhaps your character really did undergo a change of heart, but they didn't make you demonstrate it for very long. Even if you weren't pulling the wool over their eyes, others could.

Letting people off so easily for accepting magic in their past could encourage people to use yellow flowers and the like, knowing they can stop it at a later stage in their life.