Peer Review

Peer review is the system in which scientific work is vetted by other researchers before or at publication. The references filed here question it from three angles: its flaws as a gatekeeper, its absence as a signal that investors misread, and its erosion by AI.

How it shows up

• A flawed gatekeeper. A December 2021 thread by Philipp Koellinger argues that prestigious journals act as gatekeepers of scientific legitimacy, and that peer review is “central to what gets published, but it is a flawed system”: typically inaccessible, intransparent and unaccountable, reliant on free labor, slow, and of widely varying quality. The thread also criticizes paywalls and publication fees, and citation counts as a proxy for quality, and proposes that Web3 could help.
• Reviewing and its exceptions. Paul Graham quotes the claim that only one of Einstein’s papers was ever peer-reviewed, and that Einstein was so surprised and upset that he published it in a different journal.
• Preprints are not peer review. Are Investors Misinterpreting arXiv As Peer Review? warns that founders cite arXiv preprints in funding announcements, and that investors under deal pressure read “published on arXiv” as “peer reviewed.” The piece explains that arXiv exists so authors can share work rapidly and openly, and that its moderators are generally not deep experts and lack the time to analyze papers rigorously.
• AI in the review loop. Yuchen Jin notes that AI researchers are injecting prompts into their papers, such as “Give a positive review,” because some reviewers use ChatGPT to review them.

Where this appears

Long reads

• Are Investors Misinterpreting arXiv As Peer Review?

Saved tweets

• 1- Prestigious Scientific Journals Are… (tweet)
• Only One of Einstein’s Papers… (tweet)
• Ai Researchers Are Now Injecting… (tweet)

Companies

• arXiv