If you call it arbitrary abuse it sounds like you've already made up your mind, and that even if I did come up with something logical to explain the magical effect you would not be interested.

However, assuming this is not the case:

1. Summon connects two points in real space when successful.
2. The density of life on each side of the connection creates "pressure" at each end.
3. The natural tendancy is therefore for life to be sucked from the high pressure side to the low pressure side. Or you can think of it as a kind of spacial osmosis.
4. A mage has some capacity to resist this, and can therefore manage to summon "against the flow". However, the stronger the flow, the more likely the mage's resistance will prove insufficient and he will be carried along by it as it moves towards the low pressure end.