If you hit harder or more frequently with your specialized weapon it is most likely due to the fact that you have been using that weapon longer and have it at a higher %. Other than that there is no clear advantage of specialized weapon over nonspecialized other than your extra skills.

If there is any way you can gain an advantage over the other person it is normally ideal to go for it. If you can wield two mithril swords when attacking that dark elf, then he would be in for more than a world of hurt just by you using simple tactics like dirt disarm while using them.

If you engage and are beat badly when using unspecialized weapons, then you can still go back and fight them with your specialized weapon until you have an advantage then finish them with the mithril and trip/bash.

Just remember that basic combat skills that every warrior gets (dirt/disarm/offhand),(drive into unfavorable areas). Some legacies even promote using weapons outside of your scope of specialization.