...so this race would be like minotaurs?

Also, someone mentioned you wanted to draw them from CF lore?

My idea: Why not just make a race that is basically human/half-elf, set apart from CF, maybe tie in Silent Tower (they are escaping from the tower? they were trapped in the Tower...my knowledge of Silent is small) or the Ma'rin, add some runic magic to them (inherents?) or some type of drawback.

I don't know, I think Centaurs/Halfling would be your most viable options but halflings at the least wouldn't be goodie only.