I think it would be fair with bards, and necromancers, if the sleep timer would stay after you get woken up. They already had their one chance to sleep you, so let them wait the timer out to try again.