The New York Times recently had an article about how rewards influence student behavior. It basically revolves around the debate of whether extrinsic rewards can diminish intrinsic rewards. The fear is that, if students are used to getting prizes for doing well in tests, then their only purpose in doing well is to get those prizes. When the prizes are no longer given out (as in real life), then students don't work as hard. Another problem that was presented is that even if students are getting prizes, students getting a small prize might feel that he was punished, comparatively, and therefore have less motivation to try as hard.
Before I take a side in this debate, I want to say that prizes can sometimes be extremely helpful. Peter Diamandis, the founder and chairman of the X Prize Foundation, gave a talk about prizes as motivation at the Long Now Foundation, from which I've quoted before. A summary is also available. His basic thesis is that having a substantial prize encourages competition, and the aim of getting that prize motivates the economy as a whole to spend more than that amount. The result of that extra spending is a growth spurt in that industry. This was done with commercial flight, and more recently as a result of Diamandis' work, space tourism.
How, then, will the introduction of prizes as rewards for students work? Diamandis stressed that prizes need to be "at the intersection of audacity and achievability". In principal, it seems that prizes will have the same effects on students as on market competitors. Students will work harder to get the prize, thus increasing the output of the classroom (the "market") at a much lower cost. And I think this is the case.
The problem is that, unlike industry where once the technology develops everyone benefits in the long term, in the classroom both the students and the environment change, and only benefit in the short term. The team that won the Ansari X Prize will continue to have funding for their work, because now they have caught the attention of the media, the government, and various other groups. For a student who gets the highest score in a test though, there is no similar future motivation. Once the test is over, the motivation is lost, and the student goes back to the level he was pushing himself before.
Then there's the problem of the overjustification effect: when people are given external rewards for doing something, this reduces their internal motivation to do that. For example, let's say for participation in a reading program, one student is given some money as a reward. Another reads the same books outside of the program, and gets no reward. Who do you think enjoys the reading more? Probably the latter child - the first one comes to believe that he is reading for the money and not for enjoyment.
Why doesn't this happen with the X Prize? Because the teams chose to enter that competition. If I told a group of engineers that they have to build an orbit capable aircraft, they probably won't be too happy, even if they are astronautical engineers. The very purpose of the X Prize is to lure people who may not otherwise be interested in a problem. This is a very intuitive effect, and yet in schools this is often ignored. Everyone in the class is competing for the prize, regardless of whether you want to or not.
So how to best use prizes? First, they should only be used when the motivation is not high enough. If a student likes to read, there's no reason to reward him - his enjoyment from reading is enough to keep him at it, and rewarding him risks making him lost interest. Second, when prizes are offered, they should be voluntary. Sure, there may be students who are not motivated and won't join the competition, but if they are not impressed by the reward they're not going to try hard even if they were in the competition. The prize is to help motivate those on the border, who, given just the right nudge, might become intrinsically motivated.
There is one final facet I want to address. The prizes should be for tangible and useful results. More specifically, prizes should not be used for tests. With the debate on going about whether the SATs and GREs measure real education or test taking skills, it would be a great mistake to reward someone for being an excellent test taker. Tests are only proxies to measure the amount of learning. The prizes should instead be awarded based on how much extra effort someone put in. A presentation about what you learned in class is good, but the presentation which shows the student read more outside class, did some of his own experiements, is the one worthy of the prize.