Make a list of ways in which you can beat sleep.

Make a list of ways in which you can beat blackjack (assuming a victim who can't see hidden).

The sleep list should be many times longer than the blackjack list. Some of the items on the sleep list will seem stupid, yet don't appear on the blackjack list, such as "be fighting" or "gear for spell save". Therein is most of your justification.

Additionally: I can't think of any save at the moment that's generally easier to make than the A-P sleep save. (That is, sometimes it's as easy/hard as most things, other times it's easier, and A-Ps never have spellcraft... it doesn't really get better for you as a victim than that.) If you're routinely failing that save, every spellcaster in the game in competent hands can probably tear you a new one.