This line:

You tell Olum 'Without Silja I'd have had to deal with two of you.'

as you expecting us both to show up if you had been alone... which still seems to be what Tavlin was saying.

There is also this line:

Tavlin: I have three, and the Villagers just got me sadly.

which sort of implies that Villgers got you, when really it was just Olum.