Frequently Asked Questions
Why does isitcredible.com exist?
Our website exists to provide a public good: post-publication review of the existing academic literature. Our system operationalizes Karl Popper's definition of the scientific method as "the search for and the elimination of errors in the service of truth." By identifying weaknesses in published work, we can better map the limits of human knowledge, allowing us to identify what we still need to learn.
Why "Reviewer 2"?
According to the meme, Reviewer 2 is the archetype of the bad peer reviewer: critical and pedantic in an unconstructive way, primarily interested in promoting their own work and blocking anything that challenges it. At isitcredible.com, we aim to invert that meme: our Reviewer 2 is trained to be critical but also fair, and it has no interest of its own to promote.
How is isitcredible.com financed?
We aim to finance the provision of a public good by providing a private service. For $20, you can give your own text to Reviewer 2 to read and the report will be emailed to you directly, together with editorial advice on how to respond to it and proofreading. This can help you to find issues in your own work, at the same time as you help us to discover issues in the existing literature.
To support that mission, you can also make a direct donation to help us maintain and expand the Public Archive. When you do so, you can nominate a published work to review, and we will do our best to do so.
When you use the private service, are the reports private?
Yes! The report is fully automated and no human reads your original work. The PDF you upload is automatically deleted from our server as soon as the report is emailed to you. We keep the generated review on our server for two weeks in case you find there were any technical issues. After two weeks, the report itself is also permanently deleted.
Who provides the large language models (LLMs)?
Our system uses the LLMs of the standard major providers, such as Anthropic and Google. We strictly monitor their privacy policies to ensure your work is not used to train their LLMs and is deleted from their systems after processing.
Don't LLMs do this sort of thing for free?
What you get out of an LLM depends entirely on how you prompt it. Standard chat interfaces are often sycophantic—within a few prompts, it is easy to get Claude to tell you that your paper is brilliant. The value added from isitcredible.com comes from the system of prompts that we use to produce our reports.
Does Reviewer 2 have access to the Internet?
No. We do not currently trust LLMs to reliably extract information from the live web. Our agents are restricted to their training data and the text contained within your PDF. This makes our system particularly effective at finding issues related to internal consistency, research design, execution, and interpretation.
Do human researchers contribute to the reviews in the Public Archive?
The first version of a report is fully automated. However, in revised versions of the report, human researcher assistants working for isitcredible.com can add potential issues for Reviewer 2 to consider. We always try to consult with the authors of the reviewed text before adding any potential issues.
Furthermore, whenever a report is added to the Public Archive, we try to ensure that the authors of the reviewed text are notified; we encourage them to leave a private comment if there are any errors in the report and we will correct them in a revised version.
Who is Joseph Francis?
Joseph Francis is an economic historian who lives on a hill in Wales. He is also the founder and director of The Catalogue of Errors Ltd, the company that owns isitcredible.com, where he also works as a research assistant. His own research focuses on the political economy of Argentina and the United States since the eighteenth century. You can follow him on X or read his blog, The Poor Rich World.