Adam Mastroianni - blogs on research methodology

Michelle · Aug 2, 2023

You may remember Adam Mastroianni from such threads as this (about why peer review is crap) or this (about how people believe things could be better). But he recently posted an appeal to join his secret science society (that's not really secret):

I hereby invite every curious human to do science and post it on the internet.

Ask questions, collect data, write stuff, and make it available to everyone. You should feel as free to do and share research as you would feel uploading a video to YouTube or a song to Spotify.

You don’t actually need my or anyone else’s permission to do this, but sometimes people need a little encouragement, so: come on in!

Actually, let me make that a little more urgent: Please come in, we need you.

See, scientific progress has slowed. We fund more research than ever and get way less bang for our buck. We spend 15,000 years of collective effort every year on a peer review system that doesn’t do its job. Fraudsters can publish dozens of papers before they get caught, if they get caught at all.

This is bad. Our world is full of problems, and science is the main way we solve them. We’ve got climate change, an obesity epidemic, and a lot of sad people. There are folks dying of poverty and preventable disease. Heck, we still mainly make electricity by burning dinosaur bones. This can’t be as good as it gets.
Click to expand...

It's a great follow-up piece to the peer review piece mentioned above. Among his points is one that @Jonathan Edwards has made that good science does not necessarily require a lot of money.

In fact, if you're willing to use simple methods, you actually have an advantage over professional scientists. The pros wanna look cool to their colleagues (and win big grant money from the government), so they have to use the fanciest, most advanced techniques, even when simpler stuff would do them better. That's great for you, because it means the professionals will rarely investigate important questions if they don't require giant magnets or ten thousand computer cores or whatever. Cheap ideas are just lying around for you to scoop up. So scoop ‘em, darn it!
Click to expand...

He's even set up a Discord server for people to join if you reach out to him explaining the science you're doing. Given that we have had people here off and on talk about setting up studies, this might be one such avenue to explore.

I also like that he despises the term "citizen scientist" as he finds it terribly condescending.

If you call what I’m describing here “citizen science,” I will karate chop you. I despise that phrase. All science is science, regardless of the author’s credentials. Slapping the label “citizen” on science done by people working outside of institutions is just a way of widening the moat around the ivory tower, of reinforcing the false idea that only people with PhDs and academic jobs get to do “real” science.
Click to expand...

Hutan · Jun 11, 2024

Declining trust in Zeus is a technology

The latest blog from Adam Mastroianni:

There are some stats suggesting there is declining trust in educational institutions, then

Here’s an example. Lots of people are skeptical of the medical establishment and prefer not to interact with doctors. But very, very few of those skeptics would refuse to go the hospital if they, say, fell down the stairs and snapped their leg in two. That’s because emergency departments are so undeniably good at fixing busted legs that you'd be crazy to prefer poultices and witch-doctoring instead. This is what victory looks like: a case where the experts have vanquished the cranks so thoroughly that everyone can see it.

Those successes should be way more common. Anybody who is tethered to reality by believing in empirical evidence, experimentation, testing theories, all that science stuff—they have a gigantic advantage over the people who form their worldviews by looking at goat entrails or consulting the stars or whatever. It shouldn’t even be close. If patients can’t tell the difference between you and the goat entrail people, you better tug harder on your tether and pull yourself closer to reality, because you’re embarrassing yourself. It’s like spending years practicing your Super Smash Bros. technique and then losing to a 10-year-old who just mashes the buttons.

That’s why I’m not sympathetic to the most common worry about declining trust: if the public stops trusting mainstream experts and abandoning traditional institutions, they’ll start trusting sweaty podcasts charlatans and joining terrorist cells instead. I, too, would like to beat the charlatans and the terrorists, which is why I want to do better than, “Don’t trust those guys—they lack the proper accreditation!” If that’s all you got, people shouldn’t trust you. Instead of arguing from expertise, you should use your expertise to make better arguments.
Click to expand...

It reminded me of listening to a doctor on the radio recently who was very concerned that people are spending a lot of money on (often expensive) medical tests that aren't arranged by a doctor, and then spend more money on treatments designed to fix the (often spurious, sometimes completely wrong) findings from the tests. A lot of that concern is valid. But part of the problem is that the people turning to those tests have been to the medical profession with real symptoms and didn't get any answers that genuinely helped (and were probably patronised and gaslighted as well). And I'm reminded of the so-called tests for conversion disorder - medically sanctioned approaches to diagnosis that are just as made-up as the worst of the expensive retail medical tests. The answer to the declining trust is not to shout louder 'Trust only us, we know what we are doing!'.

Arnie Pye · Jan 31, 2024

Interesting article, thank you for posting.

Kitty · Jan 31, 2024

Hutan said: ↑

The answer to the declining trust is not to shout louder 'Trust only us, we know what we are doing!'.
Click to expand...

"...and no, we're not going to explain why we're doing it. You're not in our special club."

bobbler · Jan 31, 2024

Hutan said: ↑

The latest blog from Adam Mastroianni:

There are some stats suggesting there is declining trust in educational institutions, then

It reminded me of listening to a doctor on the radio recently who was very concerned that people are spending a lot of money on (often expensive) medical tests that aren't arranged by a doctor, and then spend more money on treatments designed to fix the (often spurious, sometimes completely wrong) findings from the tests. A lot of that concern is valid. But part of the problem is that the people turning to those tests have been to the medical profession with real symptoms and didn't get any answers that genuinely helped (and were probably patronised and gaslighted as well). And I'm reminded of the so-called tests for conversion disorder - medically sanctioned approaches to diagnosis that are just as made-up as the worst of the expensive retail medical tests. The answer to the declining trust is not to shout louder 'Trust only us, we know what we are doing!'.
Click to expand...

Hmm

I'm not a big fan of the term 'better arguments'. Too ambiguous.

I think the real solution lies in them understanding that - like integrity is something that is easily lost because basically means '100% of the time sticking to truth' and that 'doing the talking' it is about stepping back and realising what their values and what they do actually are. Not what they think (or would like to think) they are.

In certain procedures the 'quality' is measured by the rate of accident or averse consequences or occurences of side-effects. In conditions deemed more important than ME it is proper outcomes ie 5yr follow-up that are measured and reported. Not through the lens of the people marking their own homework for their department. Yellow card data and statistics on which treatment works for whom/what type of issue, along with knowledge on side-effects and who certain things might tend to occur with can be incorporated to decisions on treatment or changes to it.

I think that is called 'transparency' and it corresponds to trust and integrity because someone submitted to it/is OK with that. Nothing to do with argument.

It would have been hard for a bad broken leg fixer to claim that something now at an angle visibly and definitely dodgy on an x-ray is 'perfect healing'. Could that be why the system works best there? Plus of course I'm unaware of it being a big earner area for woo, in the UK apparently it is the A&E type stuff that would never be taken on by privates' cherry-picking so I'm guessing it's expensive and harder to business-model.

The usual issue of people who think the starting point is them talking, instead of it being listening and self-effacement. Identifying where medicine itself is just a merely 'aping/feigning/performative' of what it claims to be, in the same way 'other traditions' might have copied as a sales technique.

If there are areas where they are worse or no better than whatever made-up pretend thing they want to call these 'others' then really it isn't about 'arguments' but 'acknowledgement'. Where observation happens properly and research design isn't 'framed' to get results that fit with gut-ideas of what a condition might be.

And if they believe that they/their approach can do better one day in these areas then they need to remember what those values and approaches actually require - rather than looking for something to offer that fills the gaps.

Are all of their departments 'science-based' or whatever 'good things' they believe are rolled under that term they call themselves of 'traditional'? Is it time that within this they identify the departments who are science-based vs belief-based and fill the gaps for certain conditions where no science-based is there - with people who also know and believe in proper science only, and do not rely on trying to argue their 'belief-based' can 'look like science if you do a few random correlations'?

Does this all actually need less 'argument' and more listening and proper assessment to check those saying these things aren't the ones with the point.

Eleanor · Jun 11, 2024

Unserious Science
The first section of this article by Adam Mastroianni made me think of quite a few studies that show up in the psychosomatic subforums...

https://www.experimental-history.com/p/surely-you-can-be-serious

I once saw someone give a talk about a tiny intervention that caused a gigantic effect, something like, “We gave high school seniors a hearty slap on the back and then they scored 500 points higher on the SAT.”1

Everyone in the audience was like, “Hmm, interesting, I wonder if there were any gender effects, etc.”

I wanted to get up and yell: “EITHER THIS IS THE MOST POTENT PSYCHOLOGICAL INTERVENTION EVER, OR THIS STUDY IS TOTAL BULLSHIT.”

If those results are real, we should start a nationwide backslapping campaign immediately. We should be backslapping astronauts and before their rocket launches and Olympians before their floor routines. We should be running followup studies to see just how many SAT points we can get—does a second slap get you another 500? Or just another 250? Can you slap someone raw and turn them into a genius?

Or—much more likely—the results are not real, and we should either be a) helping this person understand where they screwed up in their methods and data analysis, or b) kicking them out for fraud.

[...]

That day, in the lecture hall, none of us were serious. We were all doing a little pantomime on the theme of science. The speaker pretended to show us some results and the we pretended to think about them. As long as we all hit our marks and said our lines, the content didn’t matter—one week you’re doing “Guys and Dolls” and the next week you’re doing “Oklahoma!”, but at the end of the day, it’s all just playacting.

This was the kind of thing that eventually drove me out of academia. I love playing pretend and I love doing science, but not at the same time.
Click to expand...

Hutan · Jun 11, 2024

Click on the tag at the top left of the thread - 'Mastroianni' - for more content by Adam Mastroianni e.g.
The Rise and Fall of Peer Review, 2022, Mastroianni
[Preprint] Things could be better, 2022, Mastroianni and Ludwin-Peery

Log in or Sign up

Adam Mastroianni - blogs on research methodology

Michelle Senior Member (Voting Rights)

Hutan Moderator Staff Member

Arnie Pye Senior Member (Voting Rights)

Kitty Senior Member (Voting Rights)

bobbler Senior Member (Voting Rights)

Eleanor Senior Member (Voting Rights)

Hutan Moderator Staff Member

Share This Page

Log in or Sign up

Adam Mastroianni - blogs on research methodology

Michelle Senior Member (Voting Rights)

Hutan Moderator Staff Member

Arnie Pye Senior Member (Voting Rights)

Kitty Senior Member (Voting Rights)

bobbler Senior Member (Voting Rights)

Eleanor Senior Member (Voting Rights)

Hutan Moderator Staff Member

Share This Page

Useful Searches