Three experiments among 42 players? Were observations made among one game, or three? Even if there were three sets of 21 games, I give even money that the observed difference is caused by chance.

I’d be interested in seeing the sample repeated with 100 players, each playing six games – two controlled, two (correctly) knowing the gender, and two (incorrectly) knowing the gender – without playing the same opponent more than once.

However, the paper does not establish what is claimed in the abstract: that “…gender stereotypes are mainly responsible for the underperformance of women in chess.”

I’ll walk you through why it doesn’t, and hopefully my example will clear up some misinterpretations of stereotype threat discussed elsewhere by Paul Sackett.

We have to take a look at the experimental design, and the issues are a little delicate, but my critique of the Maass claim will take the form of a reductio ad absurdum.

Suppose that instead of comparing women and men at chess, Anne Maass decides her next project will be to compare men and computers.

Computers can be built to outperform even the top men, but suppose Maass thinks this is just due to social psychology, in that a man facing a computer gets nervous from the fact that his opponent has orders of magnitude more processing power, and consequently the man often melts under pressure, so that maybe if Gary Kasparov thought he was playing Jack Taylor instead of IBM’s Deep Blue then he would have been victorious.

Now, suppose for the sake of argument that IBM has built a whole army of chess computers, which are generally better than men, but have a certain amount of variation in their abilities. Maass thus begins her new experiment comparing men and computers, similar in design to her current experiment comparing women and men.

Step 1: Maass matches each man in her sample with a computer of equal chess ability.

Step 2: Maass divides the man/computer pairs into two groups. There is a control group where men think they’re playing against other men and an experimental group where men are truthfully told that they’re playing against computers.

In the control group where men think they’re playing men, the matching by ability means we should expect men to do about as well as their inanimate opponents. So suppose this happens.

In the experimental group where men are told they’re playing computers, it might be entirely reasonable to expect men to get nervous and perform worse than their silicon doppelgangers. So again, suppose this is what happens.

But now suppose Masse publishes that stereotypes are “…mainly responsible for the underperformance” of men against computers. This conclusion cannot follow from the (hypothetical, albeit expected) results, because the computers were designed by IBM to be much better (on average) than men. Even with a gigantic difference in average abilities, the matching means men should play about as well as computers in the control group, so this occurrence wouldn’t be at all surprising. While the computer versus human results would show stereotypes can impact performance, they of course couldn’t imply that stereotypes are mainly responsible for the gap between humans and computers, since the conclusion would be false. There could still be a large between between the average man and average computer, even when the men are falsely told they are playing other men.

Similar incorrect interpretations are often found in the stereotype threat literature for ethnic group differences in standardized tests, but instead of matched designs these studies instead usually use statistical adjustments for subjects’ previous SAT scores.

Look carefully before declaring checkmate!

Any way I’ll be subscribing to your feed and I hope you post again soon!