I just fought some imperial and he fled away.. when I followed him (north), I bumped into two centurions.. when I wanted to go south again then, it said "The centurion blocks your path with quiet menace", and the guy that I fought was not even in the room .. it would seem to me that this is a bit tricky, no ?