When you've got a unit, some gold in your pocket, and nothing to do, it is often useful to train your men. Training does the obvious: it makes them a better-trained unit, and often makes them into a slightly more cohesive force. (Adversity brings people together, after all.)
The longer you train, the more likely you will see a benefit. Training may rise and cohesion may rise. On the other hand, the equipment is bound to get banged up a bit, and your men aren't going to be too happy about spending the better part of a day hitting each other with wooden weapons. If they do this every day, they could get quite angry indeed.
Training costs a little bit of gold, for fashioning fake weapons, padding armor, and buying things like rope and cloth for making practice dummies and such. If you practice with a lot of men, or for a long time, these costs can add up a little bit, but will usually not mount to more than two or three gold per session.
In terms of combat strength, training is often a net negative in the short term -- decreases in morale and damage to equipment will offset the increased training and cohesion. Of course, morale can rise, and equipment can be repaired.
After you've reached 90% Training and 100% Cohesion then training your men will have no further effect. For costs, see the Repair and training costs article.
At the Academy
If you have come to the conclusion that some of your character's skills have gotten rusty, or are just not up to snuff, then the academy is the place to go. Tutors of varying abilities are for hire there to teach you about any subject. They are there at your convenience: you tell them how long you are willing to submit to their tutelage for the afternoon, and you select an instructor. They in turn will determine how much to charge.
Just as in any classroom, you are not paying for knowledge, you are paying for the opportunity to learn. If you fail to learn anything, that is not their fault. Perhaps you were daydreaming, or had too heavy a lunch, or just didn't understand the lesson.
Naturally, they will not give up their secrets to the uninitiated. Those tutors who know the arts of infiltration will not see fit to teach those secrets to a mere Soldier or Trader. Similarly, no bureaucracy tutor is going to waste his time trying to cram the secrets of accounting through the thick skull of a Hero. The options available to you at the academy will only be those appropriate to your current class.
The more hours you spend training and the more skilled your tutor is, the more likely you will improve your skills. The amount that you learn depends on how long you spend in the training session, and the type of tutor you hired. With a normal tutor, you will most likely only gain 1% skill per session. Long sessions with expert tutors can yield larger increases. It is possible to gain more than 1% in a single session, but it is unlikely.
Whenever you train and don't get an advance, the game nevertheless remembers that you trained. All those "near misses" add up, so those 2 hours simply pushed you over the edge towards a skill gain that you wouldn't have gotten if there hadn't been those "failures" earlier.
At the Tournament
When visiting a tournament, you can also train the skills used there, i.e. swordfighting and jousting. This training is for free, but you only get one round of training per turn and it is less effective than the academy. However, you can train beyond 80% in a tournament.
It is possible to gain swordfighting skill when dueling. This can only occur when dueling to Surrender or Death. Dueling is inherently dangerous and can lead to wounds, which can have long term negative impact on your characters, or even death.
All skills also improve with experience. Of course during combat your unit will determine what kind of skills you can train. Commanding an archer unit does not give you much opportunity to train jousting.