The latest scoring screw-up offers a golden opportunity to find out just how predictive -- or biased -- the controversial test really is.
For thousands of high school students, the recent news that there were scoring errors in thousands of this year’s Scholastic Assessment Test scores is the stuff that nightmares are made of — particularly in this nervous time of envelopes arriving with colleges’ rejections and acceptances. And the number of students affected isn’t minimal. The College Board now admits that about 4,600 students — or almost 1 percent of the 495,000 who took the October 2005 test — had received erroneous scores as a result of answer sheets expanded by moisture, as well as other problems. While not trying to minimize the anxiety and pain of these college aspirants, I must confess that for a social scientist such as myself, these scoring errors — particularly ones from years past, which we must assume also exist in significant numbers — are a dream come true because they afford an opportunity to finally subject the controversial test to an important experiment.
I have long fantasized about persuading New York University (where I work) or some other selective college to let me into the admissions office in the middle of the night. I wouldn’t steal anything. I would merely swap about 50 random applications in the acceptance pile with 50 in the rejection pile — altering the educational trajectories of my poor research subjects forever (assuming NYU was their top choice). Then I would secretly follow these 100 students to see how they turned out several years later. Was not getting into NYU really so bad for those who “deserved” it? Did it hurt their long-term earning power? Likewise, I might ask: How good was our admissions office at picking winners? Did the “losers” secretly swapped into the acceptance pile fare worse than those who “truly” merited admission? This, by extension, would reveal a lot about how good the SAT — on which NYU and most other selective schools still rely heavily — is at predicting academic success.
Of course, I’d probably have an easier time getting approval to shock these undergraduates in some basement psychology lab than to mess with their (and NYU’s) futures by altering their admission results. And for good reason.
Yet the latest SAT scoring errors present a wonderful natural experiment. By going back to previous years and rescoring exams to detect any scoring errors, one could essentially perform the same experiment with no intentional harm done. Did the kids who had artificially low scores, thereby getting into their second or third choice for college, rather than their first, do worse in their later outcomes? Or did they perform better at these schools than would have been predicted by the (false) test results? If the answer is “yes,” that suggests that the “true” score has some validity, but if it is “no,” then we are left to wonder if more schools should follow the route of some universities in California, Texas and other states, and deemphasize the importance of the SAT in admissions.
And surely there must be plenty of students who benefited from errors in years past by receiving higher scores than they should have. These lucky ones make for experimental fodder as well: Did they end up performing just as well as their “fake score” predicted? If so, then either the test is broken, or it has a large Pygmalion effect — the name given to a devious experiment that showed that faux test scores, when communicated to grade school teachers, affected their perceptions of students’ abilities and thereby ended up affecting their “true” abilities. Either way, the results would offer another argument for getting rid of the SAT.
Then there is the perennial issue of racial, class or gender biases in the test, which evidence from bread and butter social science points to, though by no means proves. There is, for example, a substantial economic gradient, with each additional $10,000 of family income predictive of an increased SAT score of between seven and 44 points (depending on where you are on the income curve). And there are well-documented racial differences. For instance, in research published in the Harvard Education Review, Roy O. Freedle compared race differences in responses to easy vs. hard questions and found that the differences were much greater among the easier items. His interpretation was that easier items — those using more common words that can have multiple interpretations across cultural subgroups — are more reflective of bias than hard items would be. Likewise, SAT questions measuring reading comprehension showed less “bias” than vocabulary-based questions.
On the other hand, math items showed the same easy-hard dichotomy, which raises the possibility that the differences are a result of subpopulations within racial groups performing differentially well. That is, it could be that the gap between low-performing whites and low-performing blacks is greater all around than the gap between high-performing whites and high-performing blacks. This would suggest stratification within the black community rather than test bias per se.
And then there is the issue of “stereotype threat.” When psychologists Claude Steele and Joshua Aronson primed college students with negative racial stereotypes before administering achievement tests, they found significantly worse-than-predicted scores for minority students (as predicted, incidentally, by their SAT scores). This suggests that perhaps it is not the tests that are biased, but the entire culture.
But the data are far from conclusive on the subject. In the book “The Shape of the River: Long-Term Consequences of Considering Race in College and University Admissions,” authors William G. Bowen and Derek Bok (former presidents of Princeton and Harvard, respectively) show that minority students admitted to elite colleges with lower SAT scores than their white counterparts go on to perform equally well post-graduation on a variety of measures, including professional achievement and community service. On the other hand, these students do not fare as well in other important measures, such as income.
The proposed experiment with SAT scoring errors would certainly shed light on the issue of race, class or gender bias as well. If the error-induced score gaps were equally predictive of later success for specific groups, then they would suggest little bias in the SAT. If they were more predictive for nonwhites — that is, if the “true” scores were less valid — then they would suggest significant bias. And, of course, we must be open to the counterintuitive possibility that such results would show that the test in its current form actually helps minority students. (In fact, this was the original, lofty dream of the SAT — to counteract the old-boy, networked admissions game for elite schools with a universal, meritocratic standard.)
Oh how I would love to get my hands on these data!
And before you call me completely selfish, I must say that, just maybe, the news for this year’s seniors isn’t all bad. If you get a rejection letter from your dream school, you can always blame it on a scoring mistake — you might have been Yale-bound, if it weren’t for the darn College Board tests. At least, that’s what you can now tell your parents as they pack you off to state college. Sometimes, even in the world of education, ignorance is bliss.
This story has been changed since it was first published.
More Related Stories
- The best of Tumblr porn
- From global warming to fluoride: Why do people deny science?
- What does it really feel like to fall out of a building?
- How Dan Savage lost it
- Is the Environmental Defense Fund ruining environmentalism?
- Nancy Jo Sales on L.A. celeb robbers: "The Bling Ring kids were depressed"
- “Arrested Development,” hurry up and get here so you can stop being so annoying
- Top 5 investigative videos of the week: "Winning" Afghanistan
- Will U.S. amphibians become endangered species?
- Must-do's: What we like this week
- Josh Ritter makes his "Blood on the Tracks"
- I don't hate millennials anymore!
- Illinois' fracking and coal rush is a national crisis
- Jester clowns Westboro Baptist Church
- GOP: Party of crybabies
- What's 2013's "Gone Girl"? Here are this summer's best reads
- Twitter talks back: Obama's missed salute
- Fox executive behind "Does Someone Have to Go?" leaving the network
- Developers evict historic women's shelter to build luxury hotel
- Guantánamo prisoner on hunger strike cries for help on Twitter
- Kaitlyn Hunt refuses plea offer, will go to court over high school relationship
Featured Slide Shows
The week in 10 picsclose X
- 1 of 11
Lisa Montgomery embraces her nephew Thursday after a tornado tore apart her home in Cleburne, Texas. The twister killed six people and destroyed entire swaths of the North Texas town.
Credit: AP/LM Otero
Jack McMahon, the defense attorney for abortion doctor Kermit Gosnell, speaks outside the Criminal Justice Center in Philadelphia Tuesday. His client was convicted of killing three babies in his clinic, and will serve multiple life sentences.
Credit: AP/Matt Rourke
A photo taken Monday captures Vice President Joe Biden's response to a Milwaukee second-grader's innovative proposal to end America's epidemic of gun violence. This guy!
Credit: AP/Jenny Aicher
Sen. Rand Paul, R-Ky., flanked by a grouper-eyed Michele Bachmann, addresses the IRS' admission that it targeted Tea Party groups in advance of the 2012 election. In an op-ed for CNN Thursday, the Kentucky senator slammed the president for his faux outrage.
Credit: AP/Molly Riley
Ousted IRS chief Steven Miller is sworn in on Capitol Hill Friday. Miller testified before the House Ways and Means Committee on the extra scrutiny the agency gave conservative groups applying for tax-exempt status.
Credit: AP/J. Scott Applewhite
Attorney General Eric Holder pauses as he testifies on Capitol Hill before the House Judiciary Committee Wednesday. Holder is under fire, among other things, for the Justice Department's gathering of phone records at the Associated Press.
Credit: AP/Carolyn Kaster
O.J. Simpson sits during an evidentiary hearing at Clark County District Court in Las Vegas, Nev., Thursday. Simpson, who is currently serving a nine-to-33-year sentence in state prison for armed robbery and kidnapping, is using a writ of habeas corpus to seek a new trial.
Credit: AP/Las Vegas Review-Journal/Jeff Scheid
Major Tom to ground control: On Sunday astronaut Chris Hadfield recorded the first music video from space, a cover of David Bowie's "Space Oddity."
Credit: AP/NASA/Chris Hadfield
When it rains it pours. President Barack Obama speaks during a news conference Thursday with Turkish Prime Minister Recep Tayyip Erdogan, inexplicably inspiring an #umbrellagate Twitter meme.
Credit: AP/Jacquelyn Martin
A smoke plume rises high above a road block at the intersection of County A and Ross Road east of Solon Springs, Wis., Tuesday. No injuries were reported, but the the wildfire caused evacuations across northwestern Wisconsin.
Credit: AP/The Duluth News-Tribune/Clint Austin
Recent Slide Shows
- 1 of 11
Dalton Conley is a professor at and director of NYU's Center for Advanced Social Science Research. His latest book is "The Pecking Order: Which Siblings Succeed and Why."