Why the SAT is not a great measure of Intelligence

The econblog Marginal Revolution points to an interesting academic study of intelligence (PDF) published by Professors Keith E. Stanovich and Richard F. West.

In 7 different studies, the authors observed that a large number of thinking biases are uncorrelated with cognitive ability. These thinking biases include some of the most classic and well-studied biases in the heuristics and biases literature, including the conjunction effect, framing effects, anchoring effects, outcome bias, base-rate neglect, “less is more” effects, affect biases, omission bias, myside bias, sunk-cost effect, and certainty effects that violate the axioms of expected utility theory. In a further experiment, the authors nonetheless showed that cognitive ability does correlate with the tendency to avoid some rational thinking biases, specifically the tendency to display denominator neglect, probability matching rather than maximizing, belief bias, and matching bias on the 4-card selection task. The authors present a framework for predicting when cognitive ability will and will not correlate with a rational thinking tendency.

Basically, the paper reports on various experiments that purport to determine how ‘cognitive ability’ (i.e., intelligence) affects an individual’s decision making processes.  What caught my attention was the study’s use of SAT scores to determine the cognitive ability of individuals in the test group.

The participants were 434 undergraduate students (102 men and 332 women) recruited through an introductory psychology subject pool at a medium-sized state university in the United States. Their mean age was 19.0 years….

Students were asked to indicate their verbal, mathematical, and total SAT scores on the demographics form. The mean reported verbal SAT score of the students was 577 (SD 68), the mean reported mathematical SAT score was 572 (SD 69), and the mean total SAT score was 1149 (SD 110). The institution-wide averages for this university in 2006 were 565, 575, and 1140, respectively…

The total SAT score was used as an index of cognitive ability in the analyses reported here because it loads highly on psychometric g (Frey & Detterman, 2004; Unsworth & Engle, 2007). For the purposes of some of the analyses described below, the 206 students with SAT scores below the median (1150) were assigned to the low-SAT group, and the 228 remaining students were assigned to the high-SAT group. Parallel analyses that are fully continuous and that did not involve partitioning the sample are also reported.

A serious flaw in this research is the erroneous conflation of student reported SAT scores with cognitive ability. As a professional SAT tutor and educator who has individually taught many hundreds of students to achieve substantially higher scores, I do not believe that SAT performance can or should be considered an appropriate baseline measure of cognitive ability, as this study does.

Simply put, proper SAT training can improve student scores by literally hundreds of points, and while there is indeed a score ceiling that each student usually reaches, unless the study provides statistical control for whether has student has been ‘optimized’ to achieve this ceiling, its reliance on SAT scores as a test of a student’s cognitive ability must be considered fatally flawed.

Moreover, even if a student has not had any training, the College Board’s own study (PDF) shows that simply by taking the test multiple times, a student can improve his or her scores.   Without controlling for how many times a student took the test, let alone whether the student is reporting best scores in individual subjects from the same test or different tests, there is simply no way to say that a student’s final reported SAT score is a legitimate cognitive measure on which to base other experiments of biases.

It should also be noted that many other non-intelligence factors, both internal and external, also affect SAT scores. Parental and peer pressure can have a severe (usually negative) impact on student performance. Mental fatigue (often caused by student overscheduling) and physical fatigue (common among student athletes) are also factors. Likewise, a student’s maturity level (both emotional and physical) is an important variable. There are others.

In sum, the reliance on self reported SAT scores as the definitive indicator of a student’s cognitive ability skews the results of this study to such a significant degree that they must be questioned.  I’m not saying that the conclusions of Profs. Stanovich & West do not have merit; just that without a more accurate and controlled baseline of cognitive ability, there is simply no way to tell.