At hero, I kept avoid up 100% of the time (even when wielding a weapon, usually).

I rarely ever noticed a drain.

The only exception to this is if you're spamming heals or wrath in addition to avoid. But in general, if you're doing either of those two things, you're better off with a sword/shield. Use avoid when you want to play with respond and intensify.