AP nixes guessing penalty. SAT next?

Beginning in May 2011, the College Board will eliminate the ‘guessing penalty’ for AP exams.

Under the old College Board policy, AP scores were based on the total number of correct answers minus a fraction for every incorrect answer—one-third of a point for questions with four possible answers and one-fourth of a point for questions with five possible answers. AP students were trained to work the odds by eliminating one or more possible answers and then making an “educated guess.” In fact, the College Board traditionally supported this strategy saying, “…if you have SOME knowledge of the question, and can eliminate one or more answer choices, informed guessing from among the remaining choices is usually to your advantage.”

The College Board similarly applies a 1/4 point guessing penalty for each incorrect SAT multiple choice answer, so it’s not a stretch to assume that a change AP scoring may presage a change in SAT scoring down the road:

Robert Schaeffer, public education director of the National Center for Fair and Open Testing, said he viewed it as significant that the College Board was changing any policy related to guessing, since the organization has argued since the 1950s that a penalty was needed. He said he looked forward to seeing how the College Board would justify having one policy for AP and another for the SAT.

For the moment, the College Board maintains a studiously ambiguous stance on the prospects of change in SAT scoring policy:

As for the SAT, the College Board spokeswoman indicated that the change is being announced only for AP. “The SAT Program has no immediate plans to change scoring processes, and will keep the public informed if that position changes,” she said.

I wouldn’t exactly call that a firm statement in support of the existing SAT scoring system. Would you?

The sudden impetus for the change may come from the increased popularity of the ACT, which does not use a guessing penalty:

Schaeffer also said that the guessing penalty is “a major competitive disadvantage for the SAT” vs. the ACT. “While the ACT is not a better test in any psychometric sense, the lack of a guessing penalty is one of the ways it is more consumer-friendly,” he said.

Although I agree with Mr. Schaeffer that the lack of a guessing penalty most likely contributes to the ACT’s increasing popularity, I do not believe the difference in scoring policy is purely cosmetic.

The SAT’s guessing penalty distorts the test’s ability to evaluate student performance accurately because it makes the test more about evaluating a student’s level of self-confidence, and less about evaluating his or her level of actual knowledge.

With the guessing penalty in play, it’s not enough just to choose an answer. For each question, the student also has to decide whether he or she is confident enough in the choice to risk a quarter point reduction for being wrong. This extra layer of decision making tends to discourage less assertive students, who will often shy away from those questions whose answers they are not wholly sure of, including questions where they would otherwise guess correctly were it not for their fear of the guessing penalty.

The result is that the guessing penalty ends up favoring the bold, guessing student over the more cautious, selective student – exactly the opposite outcome from what the guessing penalty is supposed to accomplish.

Studies suggest that the guessing penalty may also contribute to the persistent lag in the SAT performance of female test takers (especially in Math).

Research indicates that males are more likely to take risks on the test and guess when they do not know the answer, whereas females tend to answer the question only if they are sure they are correct. Unwillingness to make educated guesses on this exam has been shown to have a significant negative impact on scores.

The ACT does not have a guessing penalty, which may be one reason why the gender gap on that test is much smaller.

In my own teaching experience, I find that female SAT students often display a greater tendency to skip questions when they are not completely sure of the answer – even when the answer they would have picked turns out be the correct one. These less assertive students lose points they would otherwise earn were there no guessing penalty to discourage them from answering – points more assertive students earn even though they may have no better understanding of why a particular answer is correct.

Bottom line: if and when the College Board finally does away with the SAT guessing penalty, it will be doing itself and its test takers a big favor – not only because it will make the SAT more ‘consumer friendly’ but also, and more importantly, because it will help SAT scoring better reflect each student’s level of academic performance regardless of his or her level of personal self-confidence.

SAT score map for Top 20 US Universities

Nancy Xiao at teachstreet sent me this cool map of SAT scores for universities listed in US News and World Report’s Top 20 ranking for 2010.

Via: SAT Prep Courses

Rankings lists always generate a lot of debate about what the ‘best’ schools really are, and this list, with its rather obvious northeastern bias, is sure to be no exception. And before anyone gets too excited, be aware that the map is based on US News & World Report’s university rankings and does not include liberal arts colleges (links to 2011 rankings).

Yet regardless of which particular universities you think deserve to be in the Top 20 list, Nancy’s teachstreet map provides a good illustration of the general level of SAT scores needed for admission to America’s elite colleges and universities.

Generally, top US schools require a minimum cumulative SAT score of around 2100 for a chance at admission, while the ‘rest of the best’ require a minimum score of around 2000 for consideration.

Thanks for the illustrating that Nancy!

Q&A with SAT Expert Dr. Gary Gruber

Before many of us were even old enough to take the SAT, Dr. Gary Gruber was already helping students improve their test scores. 30 years later, Dr. Gruber remains one of the foremost authorities on SAT & ACT test preparation, publishing more than 30 test prep books that have sold over 7 million copies.

Dr. Gruber’s test prep books include:

Gruber’s Complete SAT Guide
Gruber’s Complete ACT Guide 2010
Gruber’s SAT 2400
Gruber’s SAT Word Master
Gruber’s Complete SAT Reading Workbook
Gruber’s Complete SAT Writing Workbook
Gruber’s Complete SAT Math Workbook

Dr. Gruber was kind enough to answer a few questions about his long experience in test prep, his thoughts about the new SAT, and his recommendations for tutors and students.

How did you get started in test prep? Do you still personally train students?

When in fifth grade I received a 90IQ (below average) on an IQ Test, my father who was a High School teacher at the time, was concerned so he was able to get me an IQ test hoping I could study it and increase my score. However, when I looked at the test, I was so fascinated at what the questions were trying to assess, I started to figure out what strategies and thinking could have been used for the questions and saw interesting patterns of what the test-maker was trying to test. I increased my IQ to 126 and then to 150. The initial experience of scoring so low on a first IQ test and branded as “dull minded”actually developed my fascination and research with standardized tests and I was determined to afford all other students my knowledge and experience so they would show their true potential as I did. So I constantly write books, newspaper and magazine articles and columns, software, and personally teach students and teachers.

The College Board revamped the SAT in 2005. How has the new SAT changed from the old SAT? Do you think the new SAT is harder or easier than the old SAT?

The College Board had taken out the Analogies and Quantitative Comparisons and had included and Essay section. In the Reading section shorter reading passages and questions relating to “double-reading passages” were added. The new math section was enhanced and added items from third year college preparatory math.

What is the ‘Gruber method’ and how does it differ from other test prep methods?

The unique aspect of my method is that I provide a mechanism and process where the student internalizes the use of strategies and thinking skills and then reinforce those methods so that students can answer questions on the SAT or ACT without panic or brain wracking. This is actually a “fun” process. The Gruber method focuses on the student’s patterns of thinking and how the student should best answer the questions. I have also developed a nationally syndicated test which is the only one of its kind and which actually tracks a student’s thinking approach for the SAT (and ACT) and directs the student to exactly what strategies are necessary for them to learn. Instead of just learning how to solve one problem at a time, if you learn a Gruber strategy you can that problem and thousands of other problems.

How do you ensure that the practice questions in your books are accurate reflections of what students will see on the actual tests?

There are two processes. For the first, I am constantly critically analyzing all the current questions and patterns on the actual tests. The second process is based on the fact that I am in directly in touch with the research development teams for any new items or methods used in the questions on any upcoming test, so I am probably the only one besides the actual SAT or ACT people that knows exactly what is being tested and why it is being tested on the SAT or what will be tested on current and upcoming tests.

What percentage of test prep study time should students spend learning vocabulary words?

The student should not spend too much time on this—perhaps 4 hours at most. The time should be invested in learning the Important Prefixes and Roots I have developed and the 3 Vocabulary Strategies. The student might also want to learn the 291 words and their opposites, which I have developed based on research of 100’s of SAT’s.

What advice can you give to students suffering from test anxiety?

I find when the student learns specific strategies they see how a strategy can be used for a multitude of questions and when they see a question on an actual SAT that uses the strategy it reinforces a confidence in them and reduces the panic. They can also treat the SAT as a game by using my strategic approaches and the panic is also reduced as a result.

SAT vs. ACT: How should students decide which test to take?

The correlation happens to be very high for both tests in that if you score well on one you will score equivalently well on the other. However, the ACT is more memory oriented than the SAT. The material is about the same, for example, there is Grammar on both tests. Math is about the same except the ACT is less strategically oriented. There is Reading on both tests and they test about the same things. However on the ACT there is a whole section on scientific data interpretation (The SAT has some questions on this topic in the Math). Fortunately you don’t have to know the science subject matter on the ACT. If you are more prone to memory, I would take the ACT. If you are more prone to strategizing or you like puzzles, I would take the SAT. In any event, I would check with the Schools that you are applying to and find out which test they prefer.

What is the single most important piece of advice you can give to students taking the SAT or ACT?

Learn some specific strategies which can be found in my books. This will let you think mechanically without wracking your brains. When answering the questions, don’t concentrate or panic about finding the answer. Try to extract something in the question which is curious and/or which will lead you to a next step in the question. You will through this “processing” the question, enable you to get an answer.

What is the single most important piece of advice you can give to tutors teaching the SAT or ACT?

Make sure that you learn the specific strategies and teach students those strategies using many different questions which employ the strategy, so the student will see variations on how that strategy is used.

What recommendations can you give to tutors who want to use your books in their test prep programs?

In Sections VI and VII in the INTRODUCTION to the SAT book there are programs for 4 hours and longer for studying for the SAT. You can use this information to create a program for teaching the student.

In Sections III and IV in the INTRODUCTION to the ACT book there are programs for 4 hours and longer for studying for the ACT. You can use this information to create a program for teaching the student.

Always try to reinforce the strategic approach, where the student can hone and internalize strategies so that they can use them for multitudes of questions.

Thank you Dr. Gruber!

College Board SAT Class of 2009 Report

From the New York Times:

Average SAT scores in reading and writing declined by one point this year, while math scores held steady, according to a report on the high school class of 2009 released Tuesday by the College Board.
…
Average scores on the three sections of the SAT were 501 in critical reading, 493 in writing, and 515 in mathematics. Scores for each section of the test range from 200 to 800.
Average scores last year, for the high school class of 2008, were 502 in reading, 494 in writing, and 515 in math.
…
More than 1.5 million college-bound seniors took the SAT, the largest group that had ever taken the test.

Males continue to outperform females on Math and Critical Reading (slightly), while females outperform males on Writing.

larger here

Ethnic disparities in performance continue:

In critical reading, non-Hispanic white students on average scored 528, compared with 516 for Asian students, 455 for Hispanic ones and 429 for African-Americans. In math, Asian students averaged 587, compared with 536 for non-Hispanic whites, 461 for Hispanics and 426 for blacks. In writing, Asians averaged 520, compared with 517 for non-Hispanic whites, 448 for Hispanics and 421 for blacks.

There also remains a strong correlation between family income and SAT performance:

The average scores for all three sections of the test directly reflected students’ family wealth. Students from families with an annual income above $200,000 scored, on average, 68 points higher in critical reading than students from families earning less than $20,000 per year, with similar disparities for math and writing.

Critics of the SAT typically point to disparities like these to claim that the test favors wealthier white students, and to a certain extent these criticisms may be justified.  However, there is also another factor at work here:

An even sharper correlation showed up between students’ average scores and the highest educational attainment of their parents. Students whose parents did not graduate from high school averaged 420 in critical reading, 139 points lower than students whose parents had a graduate degree, who averaged 559.

The correlation between family income and/or race and SAT performance may be in some ways misleading.  It’s not necessarily that students are simply ‘buying’ better scores or that the test is culturally biased against minorities, so much as the parents of better scoring students tend to be better educated themselves, and therefore have developed skill sets that can be passed down to help their children perform more optimally.   Since better educated parents are also more likely to be both wealthy and white, these socio-economic discrepancies appear amplified in the SAT score disparities.

That’s not to say that factors of  race and income do not affect SAT performance, but simply that the relative impact of these factors on student success may be overstated when compared to the impact of parental education.

Why the SAT is not a great measure of Intelligence

The econblog Marginal Revolution points to an interesting academic study of intelligence (PDF) published by Professors Keith E. Stanovich and Richard F. West.

In 7 different studies, the authors observed that a large number of thinking biases are uncorrelated with cognitive ability. These thinking biases include some of the most classic and well-studied biases in the heuristics and biases literature, including the conjunction effect, framing effects, anchoring effects, outcome bias, base-rate neglect, “less is more” effects, affect biases, omission bias, myside bias, sunk-cost effect, and certainty effects that violate the axioms of expected utility theory. In a further experiment, the authors nonetheless showed that cognitive ability does correlate with the tendency to avoid some rational thinking biases, specifically the tendency to display denominator neglect, probability matching rather than maximizing, belief bias, and matching bias on the 4-card selection task. The authors present a framework for predicting when cognitive ability will and will not correlate with a rational thinking tendency.

Basically, the paper reports on various experiments that purport to determine how ‘cognitive ability’ (i.e., intelligence) affects an individual’s decision making processes.  What caught my attention was the study’s use of SAT scores to determine the cognitive ability of individuals in the test group.

The participants were 434 undergraduate students (102 men and 332 women) recruited through an introductory psychology subject pool at a medium-sized state university in the United States. Their mean age was 19.0 years….

Students were asked to indicate their verbal, mathematical, and total SAT scores on the demographics form. The mean reported verbal SAT score of the students was 577 (SD 68), the mean reported mathematical SAT score was 572 (SD 69), and the mean total SAT score was 1149 (SD 110). The institution-wide averages for this university in 2006 were 565, 575, and 1140, respectively…

The total SAT score was used as an index of cognitive ability in the analyses reported here because it loads highly on psychometric g (Frey & Detterman, 2004; Unsworth & Engle, 2007). For the purposes of some of the analyses described below, the 206 students with SAT scores below the median (1150) were assigned to the low-SAT group, and the 228 remaining students were assigned to the high-SAT group. Parallel analyses that are fully continuous and that did not involve partitioning the sample are also reported.

A serious flaw in this research is the erroneous conflation of student reported SAT scores with cognitive ability. As a professional SAT tutor and educator who has individually taught many hundreds of students to achieve substantially higher scores, I do not believe that SAT performance can or should be considered an appropriate baseline measure of cognitive ability, as this study does.

Simply put, proper SAT training can improve student scores by literally hundreds of points, and while there is indeed a score ceiling that each student usually reaches, unless the study provides statistical control for whether has student has been ‘optimized’ to achieve this ceiling, its reliance on SAT scores as a test of a student’s cognitive ability must be considered fatally flawed.

Moreover, even if a student has not had any training, the College Board’s own study (PDF) shows that simply by taking the test multiple times, a student can improve his or her scores.   Without controlling for how many times a student took the test, let alone whether the student is reporting best scores in individual subjects from the same test or different tests, there is simply no way to say that a student’s final reported SAT score is a legitimate cognitive measure on which to base other experiments of biases.

It should also be noted that many other non-intelligence factors, both internal and external, also affect SAT scores. Parental and peer pressure can have a severe (usually negative) impact on student performance. Mental fatigue (often caused by student overscheduling) and physical fatigue (common among student athletes) are also factors. Likewise, a student’s maturity level (both emotional and physical) is an important variable. There are others.

In sum, the reliance on self reported SAT scores as the definitive indicator of a student’s cognitive ability skews the results of this study to such a significant degree that they must be questioned.  I’m not saying that the conclusions of Profs. Stanovich & West do not have merit; just that without a more accurate and controlled baseline of cognitive ability, there is simply no way to tell.