Kinds of Replication: Examining the Meanings of “Conceptual Replication” and “Direct Replication, 2022, Derksen & Merawski

Kinds of Replication: Examining the Meanings of “Conceptual Replication” and “Direct Replication
Maarten Derksen, Jill Morawski

Abstract
Although psychology’s recent crisis has been attributed to various scientific practices, it has come to be called a “replication crisis,” prompting extensive appraisals of this putatively crucial scientific practice. These have yielded disagreements over what kind of replication is to be preferred and what phenomena are being explored, yet the proposals are all grounded in a conventional philosophy of science. This article proposes another avenue that invites moving beyond a discovery metaphor of science to rethink research as enabling realities and to consider how empirical findings enact or perform a reality. An enactment perspective appreciates multiple, dynamic realities and science as producing different entities, enactments that ever encounter differences, uncertainties, and precariousness. The axioms of an enactment perspective are described and employed to more fully understand the two kinds of replication that predominate in the crisis disputes. Although the enactment perspective described here is a relatively recent development in philosophy of science and science studies, some of its core axioms are not new to psychology, and the article concludes by revisiting psychologists’ previous calls to apprehend the dynamism of psychological reality to appreciate how scientific practices actively and unavoidably participate in performativity of reality.

https://journals.sagepub.com/doi/full/10.1177/17456916211041116

 

A long jargon filled paper of psychologists writing about psychology for other psychologists - v heavy going ! However there is a genuine attempt to get to grips with the problems that psychological (and by extension, psychiatric) research faces.

"The reformers’ efforts to create a scientific practice based on direct replication are characterized by attention to statistical and methodological detail; by an emphasis on rules, regulations, and administration; and by the important role of infrastructure. The statistical and methodological inadequacies and errors of the current practice have been listed in impressive detail (Forstmeier et al., 2017; Wicherts et al., 2016) and are generally seen to consist in researchers’ exploitation of so-called researcher degrees of freedom (Simmons et al., 2011) to arrive at the desired result. In every scientific study, many decisions have to be made (e.g., regarding sample size, which comparisons to test, which tests to report). Such decisions can have a great influence on the study’s results, as shown, for example, by Simmons et al. (2011), and opportunistic use of this flexibility increases the chance of false positives. The solution that is most commonly proposed is to constrain this freedom by directing the researcher to make these choices before data collection and publicly register the study design and data analysis plan that has an electronic date stamp as validation. This is called preregistration (Wagenmakers et al., 2012).5 In the related Registered Report (RR) format, a journal editor guarantees publication of a study if the preregistered study plan is reviewed positively, regardless of the eventual results of the study (Chambers, 2013). Thus, in an RR, both researchers and editors constrain their freedom in the interest of falsifiability, giving space to negative results and their publication."

my bold.

 

Psychosomatic ideologues are currently arguing that they should not have to meet any standards and that where subjective outcomes don't match objective ones, they should be allowed to use the subjective one because they prefer them and objective is actually subjective anyway, somehow. That's what this "extensive appraisal" is looking like right now. Is this just aspirational? Virtue signaling? Hopium? This is not happening, the cocaine-laced Flavor-aid is flowing like the Amazon. Or maybe an atmospheric river, since it's mostly hot air.

If anything, excessive replication of useless studies is becoming a new major pillar of this crisis. There is not a single reason for the absurdly high number of nearly identical variations of "CBT/vague therapy for X" or why pseudoresearchers like Chalder and Crawley keep getting funding to perform the same ritualistic nonsense. Or why, somehow, medicine has become infatuated with a MLM pseudocult to the point of promoting it as valid and serious.

This has very strong vibes of how we're dealing with climate change and how the profits of the fossil industry mostly match the massive public subsidies the industry receives. It's a sick joke to pretend psychology is dealing with this, the crisis is actually getting worse and there's obviously neither urgency nor even will to do any different. This isn't an issue at the margins, it's at the fundamental level and clearly no one wants to go there.

 

Psychology needs to be reined in. It is the wilding of medicine, and everyone who turns a blind eye to its ravages contributes to the pain it causes people.

 

Given psychology's many, many sins of the past (off books human subject experimenting, fake recovered memories of abuse and many others) whatever point they may make that is valid there is the very big question of: Can those who choose this new perspective on research be trusted to do no harm?

Someone on another thread mentioned some law of when a headline asks a question you know the answer is NO.

I'm concerned that would apply to this question. Psychology research even now as it stands is rife with the possibility for abuse. This will not change that. What psychology needs most is a different kind of person. Some grown-ups would be nice (apologies to those for whom this does not apply as in any group). There are too many people with an agenda. In psychology they overwhelm those who do work in good faith.

I'd like to see an article on how to clean house.

 

Link to paper, open access, https://journals.sagepub.com/doi/full/10.1177/17456916211041116

 

I've added link to the op - appologies for omission

 

Found myself chuckling as I skim read the abstract - some good then!

 

I could be wrong (I didn't have the courage to read the whole article, particularly long !), but I have the impression that CRG has already highlighted the best of its content. The latter reports what the proponents of the "direct replication" method defend (tends towards scientific rigor), as opposed to the so-called "conceptual method ", which strangely resembles what the worst proponents of BPS dangerously defend. Here is what characterizes this "conceptual method":

“ Confidence in theory” is valued over the “confidence in operationalizations” of researchers conducting direct replications (Crandall & Sherman, 2016, p. 93)."

" So relying on theory entails appreciation that theories “are formulated on a level that transcends the concrete evidence; and their validity does not rest on the outcome of one specific experimental paradigm” (p. 39). "

"... the conceptualists emphasize “trained judgment” (the crucial value of special expertise) over the faith in “mechanical objectivity” (rigorous, routine procedures) of researchers conducting direct replications. "

" Second, it is argued that behavior is sensitive to context, and this context is socially, culturally, and historically highly variable. We therefore cannot expect the same experimental manipulation to have the same effect in different circumstances. Because of this context sensitivity, failure of a direct replication is not informative. The proper way to bolster and extend a theory is by conceptual replication. Whereas proponents of direct replication present theory as constrained (by evidence), advocates of conceptual replication accord theory a more central place in research. "

And the following excerpt from the conclusion of the article seems to reflect that the reproducibility crisis in psychology could unfortunately get even more confused and not for the better (not towards more scientific rigor) ! :

" The stakes of electing one ontological perspective over the other (or others) and thus privileging one method over others are high. Alternatively, appreciating psychological research as enacting realities, and appreciating different methods as potentially producing different realities makes way for a genuinely open science, generative research programs, expanded reflection on ethics, and ultimately more richly informed, constructive scientific exchanges about the nature of psychological entities. "

* Underline and italic by me

 

The authors here seem to confuse the purpose of replication, which is not of itself to support a theory but to asses the reliability of the specific experimental evidence being replicated.

I suspect here when they refer to ‘conceptual replication’ they are actually talking about ‘convergent evidence’ often being necessary to support a psychological theory, given the real life situation may be more complex than can be demonstrated by a single experimental methodology. This confusion between replication and convergent evidence is, I suspect, part of why the BPS crew devoted to CBT and exercise continuously repeat exactly same experimental design with different patient groups or in slightly different formats, thinking this volume of evidence strengthens their theory, when in fact they have just failed to understand how unreliable their methodology is.

 

That's what scares me the most. It is not seeing, studying reality, the human phenomenon as it is that seems to interest them here, it is to construct this reality as they see it (theory takes precedence over reality).

And thank you Peter Trewhitt for the precision about convergent evidence.

 

This is from Critical Theory ideas. I have been reading about this and seen things which explain some of the psych ideas. Basically science and experiment leading to answers is seen as a Western European idea, not something related to reality because all science has to be interpreted through social ideas.

There are other ways of looking at evidence which each have there own, equal validity so the experience of a doctor is to be held as reflecting reality just as much as a properly designed, controlled trial.

What I believe is just as valid as what you believe sort of thing. I can hear it in the likes of Per Fink and objections to the guidelines.

It is very complicated so I may have got it a bit wrong but it is all about language rather than anything scientific.

 

maybe the scientific community can start getting interested in challenging this stuff? they can at least posture right?

 

there is one place reformers [incl medicine] do not meaningfully go: corruption.

there is one thing i have never seen them meaningfully accept ever: collective responsibility [not for action t4 to my knowledge. not for andropov's massive sluggish schizophrenia industry it seems -- they seemingly even gave award to the mastermind.]

there is one thing they never meaningfully do ever: change. virtue signalling? careers? -- yes.

[occurs to me that if there are any books on sluggish schizophrenia they might be instructive.]