Charge weapon can either improve when you charge a weapon, or when the weapon discharges in a fight (not sure if you must be wielding it). I found the learning rate on discharges noticeably higher than the learning rate on casting. Go get nine or ten light maces, charge them all up, then cycle through them fighting. You should see it improve faster - 4 affinity not withstanding.