Whenever you recruit a unit, there is a chance that you will also find a captain. When you recruit additional men for a unit without a captain, there is also a chance that you find one.
Captains are simply named soldiers. They add a little flavour to your unit and should be seen primarily as roleplaying tools.
There is also a small number in brackets behind the captain's name in the unit display, e.g. Ferdinand ( 2).
This is the leadership bonus this captain grants. Having a captain in your unit will raise your leadership skill in certain situations only by this many points. The captain will slowly improve his leadership bonus in combat.
Captains will gain experience in battle, and through experience can raise their leadership bonus. If you manage to keep the same captain around for a long time, his bonus might become quite useful.
As of now, a captain can not be removed, and he will only go away if you lose the entire unit (i.e. he will always be the last man to die). This might change without warning in the future. If you have lost the captain (impossible right now, but there will be ways added later) then you can recruit additional men and hope to find a suitable replacement. It is not possible to look for a captain only, or even to specifically look for a captain. Finding a captain is purely a matter of good fortune.