Here are the Warhammer Fantasy Roleplay first edition rules for awarding XP for good roleplaying:
These points are awarded to players on an individual basis and reflect how well they portrayed their character. Was the character played in an entertaining fashion according to alignment and career? There will be times when it is obvious that players are running their characters simply as extensions of their own personality, and this need not be a bad thing, but the gamesmaster must decide whether the character’s career, alignment and background mean that he or she really should be different. Give each player a rating (this is probably something you should keep to yourself), along the lines of Bad, Poor, Average, Good or Excellent, and award 0-50 EPs as a recognition of the way the character has been ‘brought to life’.
When allocating experience points for role-playing, you should bear in mind the player’s own conception of the character. For example, a player may have decided that his dwarf is taciturn and consequently have very little to say during role-playing encounters, but become very active during more action-orientated situations.
Generally, each player should receive 30 Experience Points per session for roleplaying, with some players gaining more and some less depending on the circumstances. Only those players who have impressed and amused you with their roleplaying should gain me maximum reward; conversely only those who have added nothing whatsoever to sessions should receive none You should avoid encouraging competition amongst the players – don’t always award the largest amounts to the player with the biggest mouth!
I’ve never liked rules like this for a number of reasons. One, I feel uncomfortable judging and rewarding players by how well they have entertained me. Especially since this passage suggests keeping it secret from the player. If this is an incentive system, how can it function if the player does not know for what they are rewarded experience?
That being said, I do like the idea of roleplaying XP, though I know this might be criticized by fundamentalists that believe XP should only be from treasure and monsters, preferably with more coming from treasure. If there are roleplaying XP though, I think they should be less subjective. Another flaw with awarding XP as suggested by Warhammer 1E is that in my experience such as system often leads to roleplaying caricatures rather than more balanced personalities, because caricatures stick out more. For example, a depressed character will be portrayed as moping all the time.
One idea that I have been playing around with is to allow players to select some goals for their characters, the completion of which will result in XP rewards. Something like minor, substantive, and major goals which would award 100, 500, and 1000 XP respectively. A 100 XP goal might be something like getting a hellhound pelt crafted into a suit of leather armor, fashioning a hat out of a shroom head, or transcribing looted dwarven books and donating them to the library of Ioun (all actual examples from my current campaign). Some might object that that some of these things come with their own reward (like getting a suit of armor) but the same thing is true of treasure.
The best part of this is that it seems like it would reward engagement with the setting. I’m always looking for ways that I can get players to be more self-directed. Adventure paths have trained players to just go along rather than venturing out on their own. Goals would need to be negotiated beforehand, and thus would not be arbitrary. A good goal, just like in real life, should be easily measurable. It also offloads some work from the referee to the players, which is often a good thing.