Addressing the theory crisis in psychology (2019) Oberauer & Lewandowsky

J

JohnTheJack

Moderator
Staff member
Abstract
A worrying number of psychological findings are not replicable. Diagnoses of the causes of this “replication crisis,” and recommendations to address it, have nearly exclusively focused on methods of data collection, analysis, and reporting.

We argue that a further cause of poor replicability is the often weak logical link between theories and their empirical tests. We propose a distinction between discovery-oriented and theory-testing research.

In discovery-oriented research, theories do not strongly imply hypotheses by which they can be tested, but rather define a search space for the discovery of effects that would support them. Failures to find these effects do not question the theory. This endeavor necessarily engenders a high risk of Type I errors—that is, publication of findings that will not replicate.

Theory-testing research, by contrast, relies on theories that strongly imply hypotheses, such that disconfirmation of the hypothesis provides evidence against the theory. Theory-testing research engenders a smaller risk of Type I errors.

A strong link between theories and hypotheses is best achieved by formalizing theories as computational models. We critically revisit recommendations for addressing the “replication crisis,” including the proposal to distinguish exploratory from confirmatory research, and the preregistration of hypotheses and analysis plans.
Click to expand...

https://link.springer.com/article/10.3758/s13423-019-01645-2
 
Theory-testing research, by contrast, relies on theories that strongly imply hypotheses, such that disconfirmation of the hypothesis provides evidence against the theory. Theory-testing research engenders a smaller risk of Type I errors.
Click to expand...

A.k.a. listen to Popper, and practice modern scientific methodology. Who would have thought this would be better than what is essentially verificationism, or the attempt to amass data to justify your unproven theories.

I have not yet read the full paper, I might say more later.
 
Barry said:
Combined with negligible attempts to gather data that might disprove them.
Click to expand...
I think it's worse than that. With PACE, they deliberately buried objective data disproving them. In fact this happened in many psychosocial trials for ME, including SMILE.

They know their beliefs are disproven and don't care. This is beyond not trying and cherry-picking only the good stuff, it's deliberately obscuring and hiding contradictory evidence.
 
alex3619 said:
The knowing improper calculation of "normal" for SF-36PF in the PACE trial is an example,
Click to expand...
That, and their handling of actimeters, are clear fraud, IMHO.

Also a good case for criminal charges of financial fraud, for spending large sums of public monies on research they knew was dodgy.

Absolute minimum consequences should be losing their jobs, being permanently struck of all professional registers, and never being allowed anywhere near a patient, a scientific study, or a clinical/academic/medico-legal/policy advisory position ever again.
 
alex3619 said:
A.k.a. listen to Popper, and practice modern scientific methodology. Who would have thought this would be better than what is essentially verificationism, or the attempt to amass data to justify your unproven theories.

I have not yet read the full paper, I might say more later.
Click to expand...
Well your theory so far seems sound, and I expect a read of the full paper will verify it ;)
 
alex3619 said:
I regard the actometer issue as highly probable improper conduct, but it cannot be proven beyond any doubt. The improper conduct with SD and SF36-PF can be proven beyond ANY doubt.
Click to expand...
Dropping all objective measures was unacceptable for a trial like this, no matter the rationale. As is the whole tweaking and readjusting midcourse. This was THE definitive trial, following several similar ones in which they had the opportunity to fine tune their methodology for nearly 2 decades.

Making adjustments to methodology and testing outcomes is justifiable in small prospective trials testing entirely new treatments. This was the largest comparative trial of psychological treatments in the UK. This team had combined authorships on roughly a dozen or so very similar trials, which had all the same flaws, intent and methodology so it can't be argued they were surprised by anything, they had already encountered every one of the issues they claimed justified deviating from protocol and it is all documented.

Not a single deviation should have been allowed. This wasn't entry-level, it was pro-league stuff where it was no only expected but required to have worked out all the kinks. To entirely drop the primary outcomes is beyond unacceptable, it deserves a full investigation of the entire process, how it got approval, how suck tinkering was allowed despite the "maturity" of the researchers on the same methodology, treatments and cohorts. Especially as they had already achieved promoting their treatments into practice. How could treatments made officially into practice present any surprise in a large trial by researchers with combined decades of experience on this exact topic?! Ridiculous.

And even at this point they had already run several mid-sized trials. They didn't suddenly jump from tiny trials to a very large one. They had gone through a 2 decades-long process of refinement and should have been expected to find no surprises they haven't previously encountered, having already done nearly the exact same work multiple times.

As excuses go, this is as pathetic as Uri Geller on Johnny Carson's show blaming his "magic" not working because of pressure and Carson's skepticism. Geller had been doing his spoon-bending magic for years at this point. The misconduct is clear and blatant.
 
rvallee said:
Dropping all objective measures was unacceptable for a trial like this, no matter the rationale.
Click to expand...
Yeah, but it could be put down to stupidity or politics, not scientific misconduct. Its unlikely in my view, but its an out they may exploit.

I regard objective outcome measures as mandatory for any study in which they are possible, except for small pilot studies.

I suspect they got scared when other studies, including FINE, showed patients did not improve on objective measures.

In any case its grounds for downgrading the quality of the study in any evidence based review.
 
You must log in or register to reply here.
Back
Top Bottom