As any testing true believer will gladly affirm, mandatory, standardized tests are perfect, absolutely flawless, shining examples of the test maker’s art and science, absolutely infallible in telling one all they need to know about the academic attainment, intellect and scholarly capacity of anyone taking them. And it must be so, because the companies that make and sell such tests compete in a multi-billion dollar market. Why, if the tests were less than perfect, how could the stratospheric fees paid these companies be justified?
Consider this from USA Today:
The College Board has announced that it will allow high school students who took the SAT the June 6 exam to take the October 3 exam for free, after the organization had to discount two sections of the June exam due to incorrect instructions for those parts.
The instructions on the last section of the June 6 exam incorrectly stated that students would have 25 minutes to complete the section — the correct amount of time should have been printed to reflect a 20 minute time period. Educational Testing Services —the company that provided the test booklets for all of the students taking the test that day —reported the error to the College Board.
Oh well, mistakes happen! It probably didn’t affect many kids–did it?
Approximately 487,000 students took the SAT exam on June 6, one of them being Todd Doyle, a rising high school senior from Sewickley, Penn.
‘Although the printing error regarding the time allotted for my final reading section was minor, it was enough to confuse me about what kind of questions to expect. Any student who spent sufficient time studying for the SAT should have caught that mistake,” Doyle says.
The College Board, however, is still confident that even after discounting the affected sections, it will be able to provide accurate scores to the students who took the SAT on June 6.
‘After a comprehensive review and statistical analysis, the College Board and ETS have determined that the affected sections will not be scored and we will still be able to provide reliable scores for all students who took the SAT on June 6,’ says Zach Goldberg, Director of Media Relations for the College Board. ‘We expect to deliver scores within the usual timeframe.
Well OK then. The SAT can just blow off about 1/3 of the test and the results are still perfectly valid. Uh, wait a minute… if the College Board–which by the way is a for-profit, commercial enterprise–can just wipe away 1/3 of the tests of nearly a half-million kids and still claim they’re just as good as the whole test, what does that say about the validity of the test, in part or in whole. More on this later.
Dr. Katherine Cohen, CEO and founder of Ivywise, a college admissions counseling service, believes the College Board has handled this incident poorly.
‘It is very obvious their response was not thought out and their top priority was saving face, not transparency,’ Cohen says.
No! The College Board is a private, for-profit company that makes high-stakes standardized tests for profit. They would never put saving face over transparency!
The College Board’s response was sluggish and confusing. It took them almost three days to properly address the problem and let parents and students know what would happen.’
Cohen thinks that the College Board’s announcement of this incident was so quiet in an effort to minimize backlash — both financially and socially — towards the company.
‘It is bad business financially for the College Board to have to re-administer a test for free to that many students nationwide,’ she says. ‘A quiet announcement (with vague instructions on how to re-register and by when) serves as a barrier to many students who otherwise would have liked to re-take the test for free.
Let’s take a moment to consider what’s going on here. Nearly a half-million kids paid $54.50 to take the SAT, which does not include materials and tutoring classes taken prior to taking the SAT. After taking the test, they found out, or if Cohen is right, most didn’t, that their scores are invalid, but according to the College Board really aren’t. But the College Board will let them take the test again for free, if they were among the few to hear about the screw-up. So each of those kids has to wonder if the test they received on the SAT is accurate, if the College Board is lying to them, and whether it would make any sense to take the damned thing again considering that it may be foolish to trust the College Board in the first place. Great.
Cohen also believes the incident will affect how high school students and their families perceive the legitimacy of the College Board in the future.
‘If they have mistakes like this on the current test, how can anyone have faith that there won’t be much greater room for error on new test, when glitches or mistakes are much more likely to creep in,’ Cohen says.
What Cohen said. Consider this commercial reason for the College Board’s actions:
The SAT will be introducing a redesigned exam in 2016 in the hope of slowing the ACT’s increasing popularity. According to Cohen, in 2013-2014, 1.8 million students took the ACT while only 1.7 million took the SAT. The ACT has maintained its status as the dominant college entrance exam since 2012.
And that’s not all:
The free SAT being offered in October also presents another problem: high school seniors who plan to apply to schools before their Early Decision or Early Action deadline may be forced to submit applications with their SAT score missing.
Dan Edmonds, writing at Forbes, addresses the validity of the SAT:
A printing error on the June SAT gave some students an extra five minutes on the test. But, since some proctors caught the error, other students didn’t receive the extra time.
Further complicating the issue is the fact that test-takers were working on different sections when this timing mishap occurred. (To discourage cheating, the SAT presents test sections in different orders so that students are not all working on the same problems at the same time.)
The end result is that two sections — a 20-minute reading section with 19 questions, and a 20-minute math section with 16 questions — were compromised by this error.
The College Board has spent a week trying to come up with an appropriate response. On Monday, it announced that “after a comprehensive review and statistical analysis,” it would not score the two affected sections. The organization assured students and colleges, however, that it could still ‘provide reliable scores for all students’ who took the June 6 test.
So. The SAT is supposed to represent the state of the art in analyzing the scholarly ability of incoming college freshmen. Theoretically, every question, every section of the test is vital in producing a reliable, accurate score. But that’s no biggie. Any two sections can be disregarded at any time and the test result is just as valid?
This response suggests only two possible conclusions: Either the College Board is exaggerating the scoring accuracy now, or it has been administering a test that’s about 30 percent too long for a decade.
When the SAT last changed in 2005, the College Board, at the eleventh hour, introduced an additional 14 questions on the Writing portion of the exam. These additional questions comprised the final ten-minute section of the current — soon-to-be-former — SAT. The section was added well after most of the initial work had been done to create new test-prep materials. The last-minute addition left test-prep companies scrambling to alter their materials and add a tenth section to all the practice tests they’d been busily writing.
I’ve taught SAT prep courses, which used several substantial test prep books. The students that took my course normally raised their SAT scores from 100-300 points. Does this mean I’m a great teacher? Is that why their scores increased?
Perhaps in part, that’s true, but the primary reason is that I taught them what to expect, and specific strategies useful in taking that single, very specific type of test. I also gave them considerable graded practice in taking that specific type of test. It was those factors, more than any teaching brilliance on my part, that enabled my students to significantly raise their scores. Did I increase the IQs, or make them appreciably smarter? No. I taught them how to take a specific test as effectively as possible. In that process, it became crystal clear that eliminating various sections from scoring have a significant effect on any student’s score. Edmonds continues:
Why was this section added? It was added because the College Board’s comprehensive review and statistical analysis had led the organization to realize that the new SAT wasn’t asking enough questions to produce reliable scores.
This is an important insight, and one that makes sense. In the construction of any test, a certain minimum number of questions is necessary to produce reliable, valid results, to measure what one is purporting to measure.
It’s impossible to imagine that, when the College Board realized that it had gotten the number of questions wrong for Writing, it hadn’t also done similar analyses to check the validity of the Reading and Math questions. The numbers that the College Board arrived at — 49 Writing questions, 54 Math questions, and 67 Reading questions — were presumably the results of a much more comprehensive review than the one that the organization hurriedly conducted in the past week.
So, either the College Board decided — for reasons only that organization can know — to add sections that its 2005 analysis showed were unnecessary. Or, those sections actually are needed to produce reliable scores. Has the test been too long for a decade? Or is the College Board misrepresenting the facts now?
Good questions indeed.
…the College Board is claiming that it hasn’t compromised students’ results — and in the process, it’s tacitly admitting that it has been wasting students’ time with 16 unnecessary Math questions and 19 unnecessary Reading questions for the past ten years.
In either case, the College Board is presenting clear evidence that it shouldn’t be trusted…
Quite so, though there are good reasons to believe the ACT is also less than trustworthy. More and more colleges and universities no longer require any such test in their admissions decisions, currently over 800. In fact, many colleges admit students that refuse to take mandatory, high stakes tests in high school and accordingly are not awarded the diplomas they earned in 12 years of academic effort and demonstrated accomplishment. Fair Test has lists of those colleges and universities.
Particularly since a great many college students are no longer traditional students that graduate from high school and immediately embark upon a college adventure, standardized tests make virtually no sense. It’s good to see that more and more institutions of higher learning are recognizing that standardized test scores really tell them little or nothing about the character and probability for success of anyone.
Now if that little bit of rationality can only start trickling down to the K-12 level…