Assuming the generic shield is strapped to your arm... shouldn't the following apply?

a) When you remove the shield, there's a slight lag since you have to remove the strap. (Of course then there should be lag on removing breastplates etc also.)

b) If you are too weak for your shield, you cannot lift your arm to use it. I don't think losing weapons comes from being so weak that you can't even lift the weapon, but from the fact that it's useless in your crippled state.

But #### realism... it's never brought anything good

However there should be some more strategies to dealing with shields than offhand/shieldcleave/kote/deflect shield