---
title: "Postoperative delirium meta-analyses: a method to test sensitivity to assessment timing and freque"
id: "plos-one-5-assessing-the-robustness-of-meta-analyses-to-variability-in-the-frequency-and"
canonical_url: "https://medichelpline.com/clinical-feed/plos-one-5-assessing-the-robustness-of-meta-analyses-to-variability-in-the-frequency-and"
content_type: "clinical_feed_article"
specialty: "Critical Care"
source_name: "PLOS ONE (Medicine)"
source_url: "https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0361124"
published_at: "2026-10-22T14:00:00.000Z"
evidence_level: "Journal Feed"
license: "CC-BY-NC-4.0 / Informational Use"
---
# Postoperative delirium meta-analyses: a method to test sensitivity to assessment timing and freque
## Provenance & Clinical Metadata
- **Canonical URL:** https://medichelpline.com/clinical-feed/plos-one-5-assessing-the-robustness-of-meta-analyses-to-variability-in-the-frequency-and
- **Specialty:** [Critical Care](https://medichelpline.com/clinical-feed/critical-care.md)
- **Primary Source:** PLOS ONE (Medicine)
- **Source URL:** [Original Journal Publication](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0361124)
- **Published At:** 2026-10-22T14:00:00.000Z
- **Evidence Rating:** Journal Feed
## Executive GIST (TL;DR)
- The authors developed an expert-based scoring system to rate each included study’s confidence that reported negative cases truly had no **postoperative delirium (POD)**, based on four assessment factors: days assessed, assessment scheme, times/day, and time-of-day. - Nineteen experts were invited and 17 participated; investigators used median ratings per modality to create weighted component scores and summed these to an overall confidence score for each study. - The score range attainable for a study was reported as 0.215–0.925; an added bonus (0.05–0.15) was included when studies used an additional assessment method such as chart review, based on panelist responses. - The score is interpreted as a negative predictive value (NPV) proxy; (1 − score) represents a false omission rate, i.e., the proportion of reported negatives that might be missed **POD** cases under suboptimal assessment. - To generate hypothetical event counts, the false omission rate is multiplied by the number of non-POD cases in each study arm and added to observed events, producing recalculated event rates for sensitivity analyses of pooled estimates. - The authors applied the method to two existing meta-analyses (one observational, one RCT-based) as worked examples; individual study scores in those examples ranged from 33–77% (observational) and 22–70% (RCTs) as reported by the investigators. - Investigators ran additional analyses addressing potential differential misclassification between groups, alternative factor weighting (equal weights), and power-transformations of the false omission rate to probe conservative and aggressive scenarios. - Two central assumptions are required: that underlying POD prevalence is similar across included cohorts (so NPV mapping is appropriate) and that missed diagnoses are independent of intervention/exposure unless adjusted by the alternate method described. - The team provided equations, a stepwise guide, and worked calculations in supporting material to enable replication; they state all data and equations necessary for reproducibility are included in the manuscript and supplements. - The approach produced changes in pooled effect size, heterogeneity, and precision in the worked examples, indicating meta-analyses can be sensitive to variability in assessment frequency and timing.
## Clinical Analysis & Structured Key Points
Assessing the robustness of meta-analyses to variability in the frequency and timing of postoperative delirium assessment – a novel methodological approach | PLOS One Browse Subject Areas ? Click through the PLOS taxonomy to find articles in your field. For more information about PLOS Subject Areas, click here . Article Authors Metrics Comments Media Coverage Reader Comments Figures Figures Abstract Background Postoperative delirium (POD) is a fluctuating confusional state occurring early after surgery. Given its fluctuating nature, POD detection is influenced by the frequency and timing of screening. Sparse assessments potentially underestimate the true incidence reported in individual studies, subsequently influencing the weight of studies in meta-analyses, and the results and precision of these meta-analyses. Objective To develop a method to evaluate the robustness of meta-analyses to the variation in the frequency and timing of POD assessment across included studies. Methods We developed an expert-based scoring system to rate studies according to the confidence that a patient reported not to have POD in the study truly did not, based on the reported timing and frequency of POD assessments. We describe how to use the system to generate hypothetical event rates for studies if the frequency and timing of POD assessment were optimal and how to use these rates in a sensitivity analysis of meta-analyses including these studies. We illustrate the method applying it to two meta-analyses as work examples, one of observational studies and one of randomized controlled trials (RCTs). Results In our work examples, the expert-based score for the individual studies ranged from 33–77% for the observational and 22–70% for RCTs meta-analysis. Sensitivity analyses showed a change in the pooled effect size, heterogeneity, and precision compared to the original estimates. Conclusion We present a method to evaluate the robustness of a meta-analysis of studies having POD as an outcome, when there is variation across individual studies in the frequency and timing of POD assessment. The confidence score acts as a preliminary index of the robustness of meta-analyses. Through work examples, we showed that the robustness depends on the characteristics of individual studies included. Citation: Khaled M, Shanthanna H, Thabane L, Paul J, Absalom A, Bergese S, et al. (2026) Assessing the robustness of meta-analyses to variability in the frequency and timing of postoperative delirium assessment – a novel methodological approach. PLoS One 21(10): e0361124. https://doi.org/10.1371/journal.pone.0361124 Editor: Oyelola A. Adegboye, Menzies School of Health Research: Charles Darwin University, AUSTRALIA Received: January 7, 2026; Accepted: October 6, 2026; Published: October 22, 2026 Copyright: © 2026 Khaled et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: All relevant data are within the manuscript and its Supporting Information files. We confirm that all the data required to replicate the results of our study are included in our submission. Please note that the study submitted in this manuscript proposes a new methodological approach. All equations needed for the reproducibility of this work, in addition to a step-by-step guide for replication are included in the manuscript and supporting documents. This is NOT human subject research and NO human participants were included. (No human participant data). The work examples we use are based on publicly available data from previously published systematic reviews (which were also authored by the first, last and one of the co-authors of this manuscript). Funding: The author(s) received no specific funding for this work. Competing interests: The authors have declared that no competing interests exist. 1. Introduction Postoperative delirium (POD) is an acute fluctuating state of confusion that commonly occurs in older adults after surgery [ 1 , 2 ] and is associated with adverse outcomes [ 3 , 4 ]. The reported incidence ranges from 5–50% [ 1 ]. Variation in the observed incidence is presumably mostly explained by risk variation across patient and surgical populations [ 5 , 6 ]. However, an additional factor of variation is how delirium is assessed, for how long, and how often. Even across studies adopting a systematic assessment with the same validated delirium instrument, the frequency and timing of POD assessment may vary widely [ 7 – 9 ]. The “waxing-and-waning” nature of delirium suggests that just because delirium was not evident at the time of the assessment, one cannot exclude it would have occurred at an earlier or later time. Moreover, although POD usually occurs on the first postoperative day [ 10 , 11 ], confusion can be present up to a week later. Consequently, the frequency and timing of POD assessments critically influence delirium detection, expecting more episodes to be detected with frequent and prolonged assessments [ 12 – 14 ]. For practical reasons, various studies use different assessment approaches. Inadequate assessments diminish apparent incidence and power. When studies using different assessment approaches are combined in a meta-analysis, it is uncertain how and to what extent the difference in frequency and timing of POD assessment affects the results and their interpretation. We proposed a method for evaluating how the results of meta-analyses of comparative studies with POD as an outcome are robust / sensitive to the variation in the frequency, timing and duration of POD assessments across included studies. We present the method and its rationale, then apply it to two existing meta-analyses, one of observational studies and one of randomized controlled trials [RCTs]) as examples to illustrate the application of the method. 2. Methods 2.1 Overview of the approach We developed a method to assess the robustness of meta-analyses that include comparative studies in which POD is an outcome assessed with different frequencies, timings and durations. Our approach compares the actual results of the meta-analyses with the results that would be obtained in a hypothetical scenario where all the included studies adopted optimal frequency and timing of POD assessment. To calculate the hypothetical results, the results of each included study are to be adjusted according to the extent of deviation of POD assessment in that study from the optimal frequency and timing. In the absence of definitive evidence of what constitutes optimal frequency and timing of POD assessment, as a preliminary step, we attempted to define optimal assessment based on expert opinion. We developed a scoring system to rate studies based on experts’ confidence that the negative delirium cases reported are truly negative, as per the frequency and timing of POD assessment implemented in that study. The process of score development and how to apply it in running a sensitivity analysis to assess the robustness of a meta-analysis to the variability in the frequency and timing of POD assessment across the included studies are described below. 2.2 Development of the expert-based score system Senior clinicians and researchers from around the world with a track record on POD and related methods were invited to collaborate as expert panelists in this study. We contacted 19 experts, out of whom 17 agreed to collaborate (AA, SB, FB, DC, MC, MC, DC, KM, EM, HS, RDS, DIS, TS, JS, AT, DW, MW) and two did not respond. The panelists were asked to complete a questionnaire created with a focus on the sensitivity of the frequency and timing of POD assessment (and not, for instance, on the validity of the assessment tool used), and how frequency and timing could influence POD detection. The first and senior authors (MK, MM) pre-identified four key factors related to the frequency and timing of POD assessment which can potentially influence POD detection: 1) number of postoperative days up to which POD is assessed (1–30 days); 2) scheme of POD assessment (everyday vs . intermittently); 3) number of times/day POD is assessed; and 4) time of the day POD is assessed (morning vs . afternoon vs . night). The identification of these factors was informed by the existing literature on the optimal timing of assessment.(12–14) The full questionnaire is provided in supporting material ( S1 File ). First, panelists were asked to rank the factors in order of importance in influencing their confidence in POD detection methods. We used the expert ranking to decide the weight of each factor in the calculation of the final score ( Table 1 ), which was determined by the percentage of expert panelists ranking each factor as the most important (rank 1). If the percentage of votes for the first rank were tied, we compared the sum of votes for the first and second ranks for the tied factors. Download: PNG larger image TIFF original image Table 1. Factor weights. https://doi.org/10.1371/journal.pone.0361124.t001 After ranking, the panelists were presented, for each factor, with a hypothetical scenario for alternative modalities the factor could take on in a study, assuming the other factors were presumably optimal, and were asked, for each modality, to rate their confidence (0–10 scale, representing 0–100% confidence) that a negative case in a study (no POD) is truly negative, if delirium was assessed according to that modality. To obtain a confidence scoring system, we used the median of the expert-based confidence ratings assigned by the panelists to each possible factor modality to obtain the score for that modality. For each study, the overall confidence score is the sum of the weighted scores for the individual factors (equation 1.1, 1.2) (1.1) (1.2) Finally, at the end of the questionnaire, panelists were asked if their responses would be different if the study included an additional method of POD assessment (such as chart review). Based on the panelists’ responses to that question we added a pre-determined absolute value of 0.05, 0.1 or 0.15 points to the confidence score if the percentage of panellists responding “yes, confidence increases in all scenarios” was 30–60%, 61–90% or >90% respectively. No points were added if 1) or less (if effect estimate <1) than that in the control group by a value which is explained by the effect estimate. We applied this adjustment to calculate the false negatives, and subsequently hypothetical event rates, in only one of the intervention / exposure or control group whereas the false negatives in the other group were calculated using only equations 1-3 . Depending on the direction of the association, we decided which group to apply the adjustment in to avoid overinflation of event rates beyond the total sample size of the study. For studies showing no significant association between the intervention/exposure and outcome, we assume that the proportion of false negatives to reported negatives is equal in the intervention/exposure and control groups ( equation 4) . 2.4.2 Evaluating the sensitivity of the proposed approach to the expert-based weights. In the proposed approach, we assigned weights to each factor based on the expert-based rank and pre-defining the weights associated with each rank (35%, 30%, 20% and 15%). To evaluate the sensitivity of our approach to the expert-based weights, we conducted an additional analysis using equal weights (25%) for all factors (i.e., assuming all factors are equally important). 2.4.3. Power transformation analysis. We conducted additional analyses using power transformation estimating the hypothetical false omission rate by (1-confidence score) k , where our original linear mapping corresponds to k = 1. We used values for k to represent a conservative (k = 2 – estimated false omission rate declines more steeply as confidence increases), or aggressive (k = 0.5 – elevated false omission rates persist even at moderate to high confidence scores) scenarios, repeated the analyses under each scenario (results in supplementary material S5 File ). 3. Results 3.1 The expert-based confidence score Panelists completed the questionnaire between April and May 2023. Fig 1 shows the distribution of votes for all factors across ranks 1–4 and the ranking of the four factors determined by the percentage of expert panelists voting each factor as the most important (rank 1). Fig 2 shows the median confidence scores for the modalities each factor could take on. The number of times/day delirium was assessed was ranked as the top factor influencing the panelists’ confidence, followed by the number of postoperative days up to which POD was assessed. Median confidence scores increased from one to three assessments/day; and with higher numbers of postoperative days, with a plateau after day seven. The use of additional POD assessment methods such as chart review was valued as a factor increasing confidence by 76.5% of panelists. As per our a priori criteria this resulted in a bonus 0.1 points added to the overall score if an additional POD assessment method was reportedly used. Based on the above, we calculated the attainable overall confidence score for any study could range from 0.215–0.925. Download: PNG larger image TIFF original image Fig 1. Factor ranking based on their influence on the confidence in reported no-POD events. Fig 1(a) shows the percentage of panelists voting each factor as the first rank. For factors with a tied percentages of votes as first rank, the figure additionally shows the percentages of votes as second rank. Fig 1(b) shows the distribution of votes for all factors across ranks 1 to 4. https://doi.org/10.1371/journal.pone.0361124.g001 Download: PNG larger image TIFF original image Fig 2. Median confidence scores for each scenario of POD assessment. The graphs represent the median confidence scores for each modality of each of the four factors related to the frequency and timing of POD assessment. https://doi.org/10.1371/journal.pone.0361124.g002 3.2 Work examples 3.2.1 Fully worked calculation in an individual study. We illustrate our method using an example from a randomly selected published study to demonstrate one fully worked calculation from reported to hypothetical counts ( Table 2 ). Ali et al. [ 15 ], is an observational study evaluating the incidence and risk factors of d
## Related Clinical Research

- [Preparing Hospitals for Water Shortages: A Critical Analysis](https://medichelpline.com/clinical-feed/bmj-5-water-shortages-how-can-hospitals-prepare-and-are-water-companies-doing-enough.md)
- [GEM® Premier™ 7000 hemolysis detection across HbA, HbS, and HbF in whole blood](https://medichelpline.com/clinical-feed/pubmed-42900906.md) (DOI: 10.1515/cclm-2026-1538)
- [β‑Caryophyllene (CB2 agonist) reduces lung inflammation after 5/6 nephrectomy in mice](https://medichelpline.com/clinical-feed/pubmed-42732653.md) (DOI: 10.1016/j.bbrc.2026.154579)
- [Clinicians’ Views on “Fighting the Ventilator”: Survey of ICU Practice and Gaps](https://medichelpline.com/clinical-feed/medrxiv-23-clinicians-perspectives-on-fighting-the-ventilator.md)
- [CPR knowledge and confidence gaps in Malaysian university students — peer-led workshop outcomes](https://medichelpline.com/clinical-feed/plos-one-9-knowledge-willingness-and-self-efficacy-in-cpr-a-two-phase-study-of-prevalence.md)

## Navigation
- [← Back to Critical Care Feed](https://medichelpline.com/clinical-feed/critical-care.md)
- [← All Clinical Specialties](https://medichelpline.com/clinical-feed.md)
## Medical & Regulatory Disclaimer

> [!CAUTION]
> MedicHelpline content is structured for research, educational, and professional discovery purposes. It does not constitute individual medical advice, clinical diagnosis, or treatment recommendations.
> Always verify dosing, contraindications, and regulatory alerts against official product labeling and primary regulatory sources before clinical decision-making.