All hail the SAT snafu!

The latest scoring screw-up offers a golden opportunity to find out just how predictive -- or biased -- the controversial test really is.


All hail the SAT snafu!

For thousands of high school students, the recent news that there were scoring errors in thousands of this year’s Scholastic Assessment Test scores is the stuff that nightmares are made of — particularly in this nervous time of envelopes arriving with colleges’ rejections and acceptances. And the number of students affected isn’t minimal. The College Board now admits that about 4,600 students — or almost 1 percent of the 495,000 who took the October 2005 test — had received erroneous scores as a result of answer sheets expanded by moisture, as well as other problems. While not trying to minimize the anxiety and pain of these college aspirants, I must confess that for a social scientist such as myself, these scoring errors — particularly ones from years past, which we must assume also exist in significant numbers — are a dream come true because they afford an opportunity to finally subject the controversial test to an important experiment.

I have long fantasized about persuading New York University (where I work) or some other selective college to let me into the admissions office in the middle of the night. I wouldn’t steal anything. I would merely swap about 50 random applications in the acceptance pile with 50 in the rejection pile — altering the educational trajectories of my poor research subjects forever (assuming NYU was their top choice). Then I would secretly follow these 100 students to see how they turned out several years later. Was not getting into NYU really so bad for those who “deserved” it? Did it hurt their long-term earning power? Likewise, I might ask: How good was our admissions office at picking winners? Did the “losers” secretly swapped into the acceptance pile fare worse than those who “truly” merited admission? This, by extension, would reveal a lot about how good the SAT — on which NYU and most other selective schools still rely heavily — is at predicting academic success.

Of course, I’d probably have an easier time getting approval to shock these undergraduates in some basement psychology lab than to mess with their (and NYU’s) futures by altering their admission results. And for good reason.

Yet the latest SAT scoring errors present a wonderful natural experiment. By going back to previous years and rescoring exams to detect any scoring errors, one could essentially perform the same experiment with no intentional harm done. Did the kids who had artificially low scores, thereby getting into their second or third choice for college, rather than their first, do worse in their later outcomes? Or did they perform better at these schools than would have been predicted by the (false) test results? If the answer is “yes,” that suggests that the “true” score has some validity, but if it is “no,” then we are left to wonder if more schools should follow the route of some universities in California, Texas and other states, and deemphasize the importance of the SAT in admissions.

You Might Also Like

And surely there must be plenty of students who benefited from errors in years past by receiving higher scores than they should have. These lucky ones make for experimental fodder as well: Did they end up performing just as well as their “fake score” predicted? If so, then either the test is broken, or it has a large Pygmalion effect — the name given to a devious experiment that showed that faux test scores, when communicated to grade school teachers, affected their perceptions of students’ abilities and thereby ended up affecting their “true” abilities. Either way, the results would offer another argument for getting rid of the SAT.

Then there is the perennial issue of racial, class or gender biases in the test, which evidence from bread and butter social science points to, though by no means proves. There is, for example, a substantial economic gradient, with each additional $10,000 of family income predictive of an increased SAT score of between seven and 44 points (depending on where you are on the income curve). And there are well-documented racial differences. For instance, in research published in the Harvard Education Review, Roy O. Freedle compared race differences in responses to easy vs. hard questions and found that the differences were much greater among the easier items. His interpretation was that easier items — those using more common words that can have multiple interpretations across cultural subgroups — are more reflective of bias than hard items would be. Likewise, SAT questions measuring reading comprehension showed less “bias” than vocabulary-based questions.

On the other hand, math items showed the same easy-hard dichotomy, which raises the possibility that the differences are a result of subpopulations within racial groups performing differentially well. That is, it could be that the gap between low-performing whites and low-performing blacks is greater all around than the gap between high-performing whites and high-performing blacks. This would suggest stratification within the black community rather than test bias per se.

And then there is the issue of “stereotype threat.” When psychologists Claude Steele and Joshua Aronson primed college students with negative racial stereotypes before administering achievement tests, they found significantly worse-than-predicted scores for minority students (as predicted, incidentally, by their SAT scores). This suggests that perhaps it is not the tests that are biased, but the entire culture.

But the data are far from conclusive on the subject. In the book “The Shape of the River: Long-Term Consequences of Considering Race in College and University Admissions,” authors William G. Bowen and Derek Bok (former presidents of Princeton and Harvard, respectively) show that minority students admitted to elite colleges with lower SAT scores than their white counterparts go on to perform equally well post-graduation on a variety of measures, including professional achievement and community service. On the other hand, these students do not fare as well in other important measures, such as income.

The proposed experiment with SAT scoring errors would certainly shed light on the issue of race, class or gender bias as well. If the error-induced score gaps were equally predictive of later success for specific groups, then they would suggest little bias in the SAT. If they were more predictive for nonwhites — that is, if the “true” scores were less valid — then they would suggest significant bias. And, of course, we must be open to the counterintuitive possibility that such results would show that the test in its current form actually helps minority students. (In fact, this was the original, lofty dream of the SAT — to counteract the old-boy, networked admissions game for elite schools with a universal, meritocratic standard.)

Oh how I would love to get my hands on these data!

And before you call me completely selfish, I must say that, just maybe, the news for this year’s seniors isn’t all bad. If you get a rejection letter from your dream school, you can always blame it on a scoring mistake — you might have been Yale-bound, if it weren’t for the darn College Board tests. At least, that’s what you can now tell your parents as they pack you off to state college. Sometimes, even in the world of education, ignorance is bliss.

This story has been changed since it was first published.

Dalton Conley is University Professor and Professor of Sociology and Medicine at New York University. He is currently chair of the Children and Youth Section of the American Sociological Association. A Guggenheim fellow, Conley is also the first sociologist to receive the National Science Foundation’s Alan T. Waterman Award for best young research in any field. He lives in New York City.

More Related Stories

Featured Slide Shows

  • Share on Twitter
  • Share on Facebook
  • 1 of 11
  • Close
  • Fullscreen
  • Thumbnails
    Martyna Blaszczyk/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 1

    Pond de l'Archeveche - hundreds thousands of padlocks locked to a bridge by random couples, as a symbol of their eternal love. After another iconic Pont des Arts bridge was cleared of the padlocks in 2010 (as a safety measure), people started to place their love symbols on this one. Today both of the bridges are full of love locks again.

    Anders Andersson/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 2

    A bird's view of tulip fields near Voorhout in the Netherlands, photographed with a drone in April 2015.

    Aashit Desai/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 3

    Angalamman Festival is celebrated every year in a small town called Kaveripattinam in Tamil Nadu. Devotees, numbering in tens of thousands, converge in this town the day after Maha Shivratri to worship the deity Angalamman, meaning 'The Guardian God'. During the festival some of the worshippers paint their faces that personifies Goddess Kali. Other indulge in the ritual of piercing iron rods throughout their cheeks.

    Allan Gichigi/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 4

    Kit Mikai is a natural rock formation about 40m high found in Western Kenya. She goes up the rocks regularly to meditate. Kit Mikai, Kenya

    Chris Ludlow/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 5

    On a weekend trip to buffalo from Toronto we made a pit stop at Niagara Falls on the Canadian side. I took this shot with my nexus 5 smartphone. I was randomly shooting the falls themselves from different viewpoints when I happened to get a pretty lucky and interesting shot of this lone seagull on patrol over the falls. I didn't even realize I had captured it in the shot until I went back through the photos a few days later

    Jassen T./National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 6

    Incredibly beautiful and extremely remote. Koehn Lake, Mojave Desert, California. Aerial Image.

    Howard Singleton/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 7

    Lucky timing! The oxpecker was originally sitting on hippo's head. I could see the hippo was going into a huge yawn (threat display?) and the oxpecker had to vacate it's perch. When I snapped the pic, the oxpecker appeared on the verge of being inhaled and was perfectly positioned between the massive gaping jaws of the hippo. The oxpecker also appears to be screeching in terror and back-pedaling to avoid being a snack!

    Abrar Mohsin/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 8

    The Yetis of Nepal - The Aghoris as they are called are marked by colorful body paint and clothes

    Madeline Crowley/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 9

    Taken from a zodiac raft on a painfully cold, rainy day

    Ian Bird/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 10

    This wave is situated right near the CBD of Sydney. Some describe it as the most dangerous wave in Australia, due to it breaking on barnacle covered rocks only a few feet deep and only ten metres from the cliff face. If you fall off you could find yourself in a life and death situation. This photo was taken 300 feet directly above the wave from a helicopter, just as the surfer is pulling into the lip of the barrel.

  • Recent Slide Shows



Comment Preview

Your name will appear as username ( settings | log out )

You may use these HTML tags and attributes: <a href=""> <b> <em> <strong> <i> <blockquote>