Psychology | Is It Structurally Flawed?

Psychology is often touted as a cure-all snake oil. It's a field that pulls in around £16.8 billion in public funding.

But what exactly is psychology? And what does it seek to achieve?

A textbook response to the question would state that psychology is a discipline focused on the scientific study of the behavioural and mental processes of the human species, which at first glance seems reasonable enough. But being a sceptical member of said species, I would ask, given the complete spectrum of people as a whole, inconvenient differences such as cultural, religious, environmental, socio-economic, and so on, how possible is this? How can the field of psychology possibly make universally and scientifically true judgments on the collective peoples of the world?

Not All Sciences Are Made Equal

And scientifically should be the operative word here. After all, that is how taxpayers would expect £16.8 billion to be spent. So, it may be somewhat unfortunate to learn that in the field of academia, psychology is largely considered to be a soft science as opposed to a hard one.

And the difference isn’t as innocuous as it sounds. Hard sciences work within measurable and quantifiable phenomena, utilising controlled experiments that result in a predictive accuracy capable of being expressed in precise mathematical terms. Soft sciences, on the other hand, often rely on qualitative methods such as surveys, case studies, and interviews, all of which are inherently subjective. Even a more evidence-based soft science such as archaeology relies on the interpretation of the materials, rather than definitive facts obtained from them. Case in point, how often do you hear a new theory come out about why Stonehenge was built, and what its purpose was? Ignoring, of course, the extraterrestrial ones you see on the History Channel.

The point I am trying to make here is that we are spending vast amounts of public money on something we are pretending is entirely fact-based. When it just isn’t. Because it is a soft science.

After all, psychologists aren’t analysing fixed things which they test, measure, and put into mathematical models, like say physics or chemistry. Psychologists are measuring people, and we are conscious, unique, have the ability to make choices, and unfortunately, lie. 

But Psychology Is Still Backed By Scientific Evidence, Right?

Kind of. A big issue in the field at the moment is that the data we've been led to believe is established and scientifically accurate is now being called into question, something that has been termed 'the replicability crisis'. For those not familiar, replicability is one of the cornerstones of scientific research. It is something that should be done after scientists publish a study.

In a nutshell, the publishing scientists will say we had theory X and so we designed an experiment, carried out the experiment and here is the data we got, the data will then validate or disprove the hypothesis. Other scientists in the same field should look at the study, perform the experiment to the same specification as the original, and publish their data. Which should then validate the original research or call it into question.

However, in the field of psychology, it’s beginning to look like this hasn’t been happening. One of the most demonstrable pieces of evidence of this can be seen in the work done by The Reproducibility Project, which involved a group of researchers from 41 different institutions replicating 100 correlational and experimental studies obtained from three psychology journals. They found that the effects produced in the replication study were around half the magnitude reported in the original. In fact, of the studies replicated, only 36% had statistically significant results, down from 97%. Now it goes without saying that the disparity between the two values is both huge and troubling, particularly when we are assigning vast sums of public money and trust in what is believed to be factual evidence-based treatments.

Now, this issue hasn’t formed in a vacuum, and there is no point in pretending otherwise. Unfortunately, there are deep-rooted and structural issues with the way the field applies the scientific method.

Science Is As Good As The Evidence, And The Evidence Is Only As Good As The Test

And that quiet little elephant in the room is p-hacking. Strange name, I know. P-hacking is simply the implementation of one or more strategies designed to turn nonsignificant results into significant ones.

In STEM fields, we have something called a p-value (short for probability value), which is used to determine how statistically significant results are. It requires a null hypothesis, which is essentially a default position that typically states there is no difference or effect. In other words, the thing the researcher is challenging through their experiment. Without getting too heavy into the math, a calculation is made and a p-value is obtained. The p-value tells the researcher the probability of generating a test statistic as extreme or more extreme than the one calculated. Simplified, it is the chance of seeing strong evidence against the null hypothesis by random chance alone. Large p-values indicate the data isn’t surprising and validate the null hypothesis. Small p-values signal that the results are unlikely to happen by chance, thereby allowing the researcher to reject the null hypothesis.

Now, this mathematical framework should, theoretically, ensure the production of reliable data and evidence, so long as all of the individuals using it are honest. But as evidenced by the results of The Reproducibility Project, this is not the case.

So now that we know the problem exists, the question is, just how crazy can psychology get when people can just make it up?

Psychics, ‘Feeling The Future’, And Other Psychological Mumbo Jumbo

Yeah, you read that right. Dr Daryl Bem, professor emeritus at Cornell University, published: “Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect.”. Basically, Dr Bem conducted a series of experiments to investigate whether people could predict the future. And according to the results he fed into the p-value system, people could.

Now, a quick disclaimer, this is not evidence that psychics are real. So, if you jumped the gun and gave your life savings to a man or woman in a big hat, whilst they wiggled their fingers and chanted, get it all back. Of course, much like religion, I cannot disprove psychics, but Heisenberg’s uncertainty principle.

Before we get into the reception of Dr Bem’s research, you may be wondering just how it got published, and by whom. Well, I would direct you to the genuinely esteemed Journal of Personality and Social Psychology. Presumably, the editor over there read the study and decided "yes" was the correct response.

Suffice to say, not all in the scientific community shared the editor’s enthusiasm. And instead poked more holes in the study than a sponge beaten repeatedly by a dead porcupine. Researchers pointed out that p-values could be manipulated in a number of ways, for instance, by selectively analysing data and adjusting experimental parameters.

I, on the other hand, would suggest that more definitive testing should be conducted before declaring psychics a real thing, such as getting someone to win the lottery a hundred times, or strapping them to an oil rig and seeing how effective they are as a tidal wave detector. But I’m old school.

I'll be honest, when I first stumbled upon this study many moons ago, I had assumed that Dr Bem had nobly fallen upon his sword in order to get across the structurally flawed ‘scientific’ system within psychology. You know, stir up some debate with a little reductio ad absurdum. I pictured him hunched over a desk, maniacally laughing as he pounded the keys of his typewriter, all the while thinking ‘this will be the final straw and I shall be the one to deliver it!’ But no, as of the time of writing, he has never refuted the findings of the study, leaving the unsettling conclusion that he believed them.

Publish Or Perish 

Now, I do want to balance this by saying it isn’t necessarily the fault of psychologists and researchers who sometimes, liberally, shall we say, “calibrate” their p-values. Some responsibility must fall upon the system and mindset within academia, the so-called ‘publish or perish’ culture.

The origins of the phrase date back to the 19th century, but its insidious effects didn’t really come into full swing until the 1930s. Notably, when the then Harvard University president, James Bryant Conant, began dismissing faculty members for failing to publish a sufficient number of papers.

Fast forwarding into the 1990s and onwards, we begin to see higher education leaders across the board implement ‘journal impact factors’ as the benchmark criteria in determining a scientist’s career progression. These ‘journal impact factors’ simply refer to the number of times an article in a published journal is cited within the previous two years.

Now, the knock-on effect of adopting these policies is that replication testing doesn’t occur nearly as much as it should. Researchers need to find the next new shiny thing to keep their academic career afloat, as opposed to doing the less glamorous but essential task of making sure what is put forward is tested and fact-checked, like the good folks over at The Reproducibility Project, of which there simply aren’t enough.

This allows less scrupulous researchers, who, under the desperate pressure to get published, produce, at best, disingenuous research that they hope to slip in unchallenged and under the radar. The exception is when the research is too outrageous to ignore, such as psychics.

Scientific Journals, The Ignoble Reality

All of these problems are compounded because the overwhelming majority of scientific journals are for-profit. Not in the local family-run café way, but in the overseas exploitative sweatshop way. The North of Gangnam style way.

In a nutshell, you and I will pay taxes to the government, some of which end up going towards research grants. Universities will take some of the money from tuition fees, among other sources, e.g., donations. All of this money is then combined into a pot which is used to fund academic research in terms of materials, wages, equipment, etc.

After the study is complete, the researcher writes out their findings in an article format and submits it to a journal, where, depending on how prestigious the publication is, could charge upwards of $10,000 USD. To be very clear, the researcher, whose work has largely been funded by you, the taxpayer, pays to have their work published. This is despite the fact that the journal receives the article and usually the copyright, which they then sell to others.

The article, once published, is uploaded to an online database that universities will have to pay to access. This means that after all the time, money, and effort put into the research, the university ends up paying for its publication, and in the end, a loss of copyright that prevents free access to its own research. As the saying goes: ‘Imagine a business where people pay you to take their money’.

The Structural Flaw In Psychology That Puts Us Between A Rock And A Drug-Filled Grizzly

Now it is worth noting that just because a scientist has conducted a study, there is no guarantee, or indeed obligation, for the journal to publish it. And this is often the case for scientists who retest previously published studies to ensure they were valid to begin with, hence the replication crisis. This means that scientists, because of the way the system is structured, are actively being disincentivised from retesting data, as few are willing to devote significant time and resources to their research only to be rewarded with the sage French saying: ‘nul points’.

It would be fair to say that the journal impact factors being used at institutions are tainting the research process like a corpse in a well. They leave scientists under constant pressure to generate new studies that are glitzy enough to get cited, while at the same time contending with the ruinous cost of publication, if indeed the academic journals are willing to do so.

And in a replication crisis where the only way out is to test, we may be left with an array of social sciences that aren’t worth the paper they are written on. And with a system as flawed as this, it is little wonder that some would seek to exploit it, either for personal gain or simply to keep up.

Scientific research is, at its core, running headlong into the pitch-black of ignorance. There is a lot of falling over and banging your head, and it is from these failings that we know what not to do next time. Trying to increase the efficiency of the process through pressure is akin to screaming at goats with the goal of herding them. In the end, all you will be left with is a faint and a fart.

Citations: