Set up del Methodological Review Board

Overview

The Methodological Review Board (MRB) is inspired by the TU/Eindhoven experience, with a focus on feasibility and scalability. Prior experience of Psicostat members suggests that ensuring high methodological quality in a single research project requires several weeks of intensive work by a team with diverse expertise, and that researchers often struggle with clearly defining research questions, hypotheses, analysis plans and computing statistical power.

Given these constraints and with an eye to scalability and long-term sustainability, the MRB will follow Professor Carlo Umiltà’s suggestion that its primary role should be to assist in the pre-registration process, ensuring that it is done with a sufficient standard of clarity and transparency, rather than providing in-depth support on all details of a project. This is especially relevant considering that the lack of any review process for most current pre-registrations raises concerns about their overall quality.

Process and structure

Submissions of projects to the MRB will be entirely voluntary, offering researchers a service. Board members commit to not becoming co-authors of any project they review at any stage. The MRB will mostly focus on clarity and transparency and will not impose major changes on submitted projects. However, board members will be free to provide comments and suggestions based on their expertise.

MRB reviewers will be at least three per project: one expert in statistics, one expert in psychometrics, and one ad hoc expert in psychology, preferably with expertise in the specific subfield of the research. When needed, an additional ad hoc expert may be consulted, for instance, in psychophysiological measurement or data preprocessing, depending on the nature of the project.

The MRB will also promote Open Science practices by recommending and providing guidelines on the use of open data and open code. If researchers choose not to implement these practices, a clear justification will be required. This setup and these policies ensure that the MRB remains a supportive and constructive initiative, promoting better research practices without imposing rigid constraints.

Suggested procedure

• Aided by guidelines provided by the MRB, authors identify an appropriate pre-registration template for the research project. Existing templates (e.g., those available on OSF) are preferred.
• Authors complete the chosen template and submit it to the MRB via a file-sharing platform (e.g., Google Drive, GitHub).
• One or more rounds of review and feedback with the MRB ensure clarity and transparency and provide methodological suggestions.
• Authors publicly pre-register the project on a platform (e.g., OSF); a public link is required, though an embargoed anonymized version is permitted for a specific period.
• Final check and badge award: The MRB verifies alignment with the reviewed template and, if compliant, awards an MRB badge.

An “MRB badge” — ideally verifiable through a DOI or digital signature — will be the board’s primary output. This will be publicly posted on a dedicated website, alongside the project title and a link to the pre-registration. The badge is intended to certify that a pre-registration meets a sufficient standard of methodological clarity and transparency, with clearly acknowledged limitations.

Long-term impact

The effectiveness of the MRB will be assessed by tracking the increase in the pre-registration rate among published studies over time. Hardwicke et al. (2024) estimated that the pre-registration rate in psychological research increased from 2.1% (2014–2017) to 7% (2022). Success will be defined as exceeding a 10% rate among eligible studies, and in any case achieving a net increase within two years of the MRB’s implementation.

An additional metric may include the increased rate of pre-registered studies from research groups that had previously engaged in pre-registration.