Depending on your character's role of course. It's a good idea to retrieve/defend as long as the odds aren't overwhelmingly against you. Of course, there are exceptions. Examples:

A sneaky assassin who doesn't defend, but then assassinates the person holding the item before they can down it.

A crazy barbarian who just charges in no matter what the odds (short shelf life, but fun).

A manipulative transmuter who doesn't defend, but promises his services to those who will retrive for him, and therefore gets the item back quite quickly.

A rich thief who bribes a group of people into getting back his item or defending for him. Or alternatively just steals it back.

Any path has its plusses and minuses. I would just suggest that you don't end up being a sissy, unless that's your role.