Reliability and agreement (measurement error) are fundamental measurement properties for instruments used in clinical, psychological and educational settings, affecting the interpretability and quality of research and practice. The Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were published in 2011 to improve transparency and completeness of reporting for these studies. Since then, methodological standards and related initiatives—most notably COSMIN—have advanced, and commentaries have identified conceptual ambiguities and reporting challenges that may limit synthesis and risk-of-bias assessment. This protocol describes a methods review designed to inform development of the updated GRRAS-COSMIN reporting guidelines.
The methods review has two complementary aims: (1) systematically identify and synthesise published commentaries on and evaluations of the original GRRAS publications; and (2) map recent methodological developments in planning, conduct, analysis and interpretation of reliability, agreement and measurement error studies in health science, psychology and education that should be reflected in reporting guidance.
The specific review questions are:
This protocol follows best-practice guidance for scoping and methods reviews as recommended by the Joanna Briggs Institute and other methodological sources. Results reporting will align with an adapted PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews).
To address the two aims, two complementary search strategies are applied: forward citation tracking of the original GRRAS publications in Web of Science for aim 1, and a focused systematic search across bibliographic databases for aim 2. Each search strategy is aligned to the corresponding review aim and followed by a tailored data synthesis approach.
Included sources are peer-reviewed articles, commentaries or other scholarly publications that cite one or both original GRRAS references and offer critique, commentary or suggestions related to the guideline. Exclusions are publications that cite GRRAS solely for structuring reports, withdrawn or retracted articles, textbooks, peer reviews, supplements, and conference abstracts/posters/slides without accompanying full-text publications. There are no language restrictions; non-English/German full texts will be translated and key excerpts back-translated where necessary to check meaning.
For the second aim, the review will include primary studies, reviews, commentaries, editorials, methodological papers, tutorials and guidance documents that discuss methodological aspects of the measurement properties reliability and/or agreement/measurement error. The scope covers health sciences, psychology and education. Publication date is restricted to 1 January 2015 through 30 June 2026. Supplementary key references identified by the author team may be added even if they fall outside the date restriction, provided their inclusion is transparently documented and directly relevant to the review questions.
Aim 1 uses Web of Science for forward citation tracking of the two original GRRAS guideline publications. Aim 2 uses a multi-database approach: Embase and MEDLINE via Ovid, PsycInfo, ERIC and CINAHL via EBSCOhost. The Charité Library will be searched for relevant books and book chapters, and supplementary sources identified by the author team will be included and documented. Reference lists of included articles will be screened using the SpiderCite tool from the SR Accelerator.
For aim 1 a Web of Science forward citation search for the original GRRAS articles will be conducted. A pilot search reported in the protocol identified hundreds of citations for each original reference. For aim 2, a detailed search string was developed in MEDLINE via Ovid and will be translated for use in the other databases; the time window is 1 January 2015 to 30 June 2026. Supplementary sources deemed seminal or conceptually important by the author team will be added and documented; these are not necessarily subject to the date restriction.
Retrieved records will be exported in XML format and managed with the SR Accelerator platform. Duplicates will be removed before screening. Two reviewers will independently screen titles and abstracts, then independently assess full texts against eligibility criteria. If full texts are unavailable via standard routes, the team will contact the library or corresponding authors (two contact attempts with a two-week follow-up). Disagreements will be resolved through discussion and, if needed, a third reviewer will adjudicate. Reasons for exclusion at full-text screening will be documented.
The review team will also screen for concerns about journal or publisher practices using the Think. Check. Submit. checklist; sources raising substantial concern will be discussed before final inclusion decisions.
The first author will extract descriptive information from included full texts: title, authors, publication year, country of corresponding author, publication type, language, and stated research question or topic. Each source will be classified by field or discipline using the OECD Fields of Science categories and, for Web of Science indexed items, WoS subject categories mapped to OECD categories. For non-WoS sources (e.g., books, guidance documents) classification will be based on content, journal or book scope and author affiliations. Multiple fields may be assigned where appropriate.
Included articles addressing GRRAS commentaries (aim 1) will be analysed using descriptive content analysis. Methodological literature identified for aim 2 will be analysed using a ‘codebook’ thematic analysis approach. Findings will be presented narratively and supplemented with visuals and tables where appropriate. The synthesis will explicitly aim to identify topics and methodological developments relevant to reporting items for the planned GRRAS-COSMIN guideline, including planning, conduct, analysis and interpretation aspects of reliability and agreement/measurement error studies.
Strengths of the protocol include alignment with Joanna Briggs Institute guidance for scoping reviews, use of two complementary search strategies tailored to specific aims, no language restrictions, and a comprehensive multi-database search. Limitations acknowledged are focused search strings that may increase precision but reduce sensitivity—particularly for methodological literature inconsistently indexed or published outside the targeted databases—and that grey literature will not be searched systematically, although supplementary sources will be transparently documented.
This review uses publicly available materials and does not require ethical approval. Results will be disseminated through open-access journal publication and conference presentations. Reporting will follow an adapted PRISMA-ScR framework. The ultimate purpose of the methods review is to inform development of the GRRAS-COSMIN reporting guideline, supporting harmonisation with contemporary methodological standards and related initiatives such as COSMIN.