Among the more recent educational fads is the idea that anything “researched-based” is brilliant, unassailable, and therefore, worth tens, even hundreds of thousands to implement in any school district, thereby magically improving student achievement by previously unimaginable margins. The term is commonly used to pass off previously failed bad ideas, by changing a few terms, adding a few new acronyms, and claiming to have conducted a study proving beyond any possible doubt the efficacy of the retreaded, failed idea.
The first time I was presented with a research based, brilliant new/old scheme, I asked a number of impertinent questions, like “do you have the data sets and methodology of the study?” I was told that people like me didn’t need to know that kind of advanced information and wouldn’t be able to understand it anyway, so “shut up,” they explained. That pretty much said it all about the brilliance and validity of the “study” and the scheme, which was as useless–and as damaging to children–as one might imagine. Back in August of 2018, I wrote about this over-reliance on poorly done or faked studies in Education: But It’s Research Based:
This is also what I was talking about. Classrooms are the very definition of uncontrollable variables. Take Mrs. Smith who teaches 10thgrade English. She’s a dedicated and skilled teacher, but the seven sections she teaches every day will be very, very different because the personality of every class will be very different. The kids have a choice. They can be highly self-motivated learners, or mere oxygen processors. They get to choose, every day, how much they want to learn and retain.
Any kind of study that might have any validity is going to require years, not weeks or even months. The variables cannot be controlled in any meaningful way, and the experiment cannot be supervised by those running it. They can’t be in every classroom, every day, for years. There’s no profit in that. What happens when teachers following the brilliant theory/protocol in Smithville High School produce wonderful results as the people pushing the theory said they would, but the teachers of Jonestown High School don’t? Are the Jonestown teachers dumb? Didn’t they follow the protocol properly? What’s wrong with those kids? It has to work because it’s research-based!
A dedicated, capable teacher working for many years is, in effect, conducting research every day. But what could they possibly know? They’re only teachers! Better to spend billions on brilliant ‘new’ ideas that attempt to revitalize the same old bad ideas. It’s research-based, you know!
An example of that kind of child-exploiting research is reported by Jeff Jacoby.com:
THE TITLE of the study, published in the journal Current Biology in November 2015, was on the dry side: ‘The Negative Association Between Religiousness and Children’s Altruism across the World.’ The text was even drier. But the findings, by University of Chicago neuroscientist Jean Decety and his colleagues, proved irresistible to journalists, who gave them wide play under headlines likely to grab readers’ attention:
“Religion doesn’t make kids more generous or altruistic, study finds” (Los Angeles Times)
“Are religious children more selfish?” (Slate)
“Study: Religious Kids are Jerks” (Daily Beast)
Scores of media outlets reported on the Decety paper, which compiled data from more than 1,100 children in the United States, Canada, China, Jordan, Turkey, and South Africa. The researchers measured the generosity of the young children by giving each one a batch of attractive stickers, then inviting him or her to give away some of the stickers to other children. The more stickers the kids shared with others, the more altruistic they were deemed to be. According to the published results, the children of religious parents were less likely to share their stickers than children from non-religious households.
Notice that this “research” proceeds on an entirely faulty premise. It extrapolates from a single situation contrived by the researcher’s, attitude, beliefs and behaviors, and further tries to entangle them with religious faith. It is not only the virtual definition of “confirmation bias,” but confirmation bias produced before the “study” was ever implemented. It produced precisely the results the “researchers” wanted because it was designed to do just that.
The study, concluded The Economist, suggested ‘not only that what is preached by religion is not always what is practiced . . . but that in some unknown way the preaching makes things worse.
Not only that, it uses this entirely contrived “research” to condemn faith. Fortunately, rational people understood the lie:
Azim Shariff, a professor of social psychology, was perplexed by Decety’s analysis, which seemed to fly in the face of considerable evidence that religious people tend to be more likely to exhibit altruistic, charitable, and pro-social behavior. In his 2006 book Who Really Cares, the public policy scholar Arthur Brooks, citing years of social-science data, showed that religious practice correlated with higher rates of volunteering, donating blood, aiding the homeless, and giving money to secular charities.
The media would have us believe that the only truly virtuous people are Democrats/Socialists/Communists because they constantly virtue signal their great compassion and concern for the poor and downtrodden. Thus is the great misconception that D/S/Cs give far more to charity than evil normal Americans spread. The connection between this and faith is that it is normal Americans, not D/S/Cs, that tend to embrace religion, particularly traditional, not LGBTQWERTY-centric religion. The truth is normal Americans give far more to charity, and by a huge margin, than D/S/Cs. This is so because D/S/Cs believe it’s government’s job to provide for every need–with other people’s money. Normal Americans freely give of themselves, not being handicapped by such a belief. They don’t wait for government; they see a need and they handle it. This is true altruism.
To better understand how Decety’s team had come to such a strikingly different result, Shariff asked to see their data. When he re-crunched the numbers, Shariff discovered a major blunder: The six countries in the study had been coded by number — 1 for the United States, 2 for Canada, etc. — and in calculating the global results, the researchers had inadvertently treated those country codes as mathematical variables. Needless to say, that significantly skewed the study’s results. When Shariff re-analyzed the data without the coding error, the surprising findings vanished.
Current Biology published Shariff’s correction of Decety’s statistics in 2016. But references to the original paper continued to appear in the popular media. The Independent cited it in a 2017 article on the virtues of an atheist upbringing, for example. Buzzworthy ran a story on it (“Could Religion Actually Make Children Less Generous?“) just two months ago.
Yet any reader who looks up the study in Current Biology will find the word ‘RETRACTED’ stamped across each page in large red letters. In August, the journal formally disavowed the paper at the request of the authors. The study is now preceded by a note in which Decety acknowledges the fatal error discovered in his group’s analysis. ‘We feel it necessary to explicitly correct the scientific record,” he writes, “and we are therefore retracting the article. We apologize to the scientific community for any inconvenience caused.
In this case, the system worked as it was supposed to work. Decety willingly provided his data sets for review, and when he realized he was proved wrong, asked that the paper be retracted. One might reasonably believe this research driven by a political viewpoint, but Decety did the right, scientific, thing. This is, sadly, all too rare in our contemporary politicized “science.”
It is also almost unheard of in education “research.” The research upon which so many “research-based” systems or methods is based is terribly shoddy, but is seldom challenged because to conduct a proper study to refute it would take far too long, cost too much and involve a bewildering number of variables that could not possibly be controlled. In fact, where human beings are concerned, it’s damned difficult to replicate results, thus is such “research” practically non-falsifiable, which is what makes it so profitable, and destructive.
Let’s say we have revised an old, failed system for passing mandatory, high stakes tests. How do we prove it’s brilliant and amazingly effective? We talk a school district into letting us experiment on 10 English classes in a local high school. Using the system, we drill them relentlessly for a month or so, and if their scores improve a bit over last year’s scores, we proclaim it “research-based” and make a fortune. Actually, the researchers normally just sort of supervise. They “train” the teachers involved in their method, the school district forces them to use it, and the researchers reap the results and profits.
The problem is we can’t control any of the variables. Each test, each year, is different, requiring different responses. A poorly written essay prompt will dramatically reduce the scores across a state, as will poorly written questions of any kind. Each class of students, each year, is different. If even 5% of the students don’t much care how they do on the tests, the results will be significantly affected. Even the distribution of intelligence in various classes will have an enormous effect on the outcome, and so will scholarly ability, determination, and individual interest entirely apart from IQ.
We know what works: competent, dedicated teachers providing the best possible educational opportunity, and interested, hard working kids who faithfully do the practice required of them produce educated, capable adults. Put them in a school where adults are unquestionably in charge, and where nothing is allowed to interfere with learning, and lo and behold, kids learn! Human beings, circa 2019, learn and achieve in exactly the same ways the ancient Greeks learned and achieved. Times, technology and tools change, people don’t.
I don’t suggest that we do not develop new knowledge and tools, merely that human nature doesn’t change. We must, if we’re wise, view anything in education carrying the “research-based” label with appropriate—substantial—skepticism, unless, of course, we don’t much care what kids learn or how they’ll function as adults.