Think of it in simple terms. The higher your percentage is in a given weapon skill, the more you will hit your opponent using said weapon. Not too sure if it has a huge effect on how hard you hit, but I haven't noticed a significant difference - someone please correct me if I'm wrong on that.

Also, a good thing to know is that you will parry better against weapons you have a high proficiency in, especially as opposed to a weapon you do not have in your skill list. As an example, an assassin does not know how to use axes, so that axe-wielding warrior is gonna hit you a lot more than you'd like. That being said, it's probably a good idea to memorize which classes know which weapons and wear an appropriate weapon when possible. Warrior opponents are always a crap shoot because they potentially will know all weapons (until you're familiar with him!).

There's probably a lot more to be said on this topic, but those are the basics as far as I know...