Sensitivity Analyses for Meta-Analyses Involving Multiple Readers per Study

Jonathan Treadwell; Matthew Mitchell; Kelley Eatmon; Jane Jue; Hanna Zafar; Ursina Teitelbaum; Karen Schoelles

Appendix ESensitivity Analyses for Meta-Analyses Involving Multiple Readers per Study

Some comparative accuracy studies reported data separately for different readers. Our primary analyses discussed in the main report only used data from reader 1 for each such study. This appendix contains the results of sensitivity analysis of this choice, for two meta-analyses:

MDCT versus MRI for diagnosis (a seven-study meta-analysis in which four of the seven studies reported multiple readers separately). Of the four studies, two reported three readers each, and two reported two readers each. Thus, we performed 35 sensitivity analyses ((2x3x2x3)-1 primary analysis).
MDCT versus MRI for assessment of metastases (a five-study meta-analysis in which one of the five studies reported three readers separately. Thus, we performed 2 sensitivity analyses (3-1 primary analysis).

Sensitivity Analysis of MDCT vs. MRI for Diagnosis

The primary analysis yielded estimates for MDCT of 89% for sensitivity and 90% for specificity, whereas the estimates for MRI were 89% for sensitivity and 89% for specificity. The table below lists the results of the 35 sensitivity analyses; all analysis provided estimates that were very similar to the primary analysis.

Table E-1

Sensitivity analysis of MDCT vs. MRI for diagnosis.

Sensitivity Analysis of MDCT vs. MRI for Assessment of Metastases

The primary analysis yielded estimates for MDCT of 48% for sensitivity and 90% for specificity, whereas the estimates for MRI were 50% for sensitivity and 95% for specificity. The table below lists the results of the two sensitivity analyses; both provided estimates that were very similar to the primary analysis.

Table E-2

Sensitivity analysis of MDCT vs. MRI for assessment of metastases.

Sensitivity Analysis of MDCT vs. MRI for Assessment of Metastases

Table E-3

Sensitivity analysis of MDCT vs. MRI for assessment of metastases.

Publication Details

Copyright

Publisher

Agency for Healthcare Research and Quality (US), Rockville (MD)

NLM Citation

Treadwell J, Mitchell M, Eatmon K, et al. Imaging Tests for the Diagnosis and Staging of Pancreatic Adenocarcinoma [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014 Sep. (Comparative Effectiveness Review, No. 141.) Appendix E, Sensitivity Analyses for Meta-Analyses Involving Multiple Readers per Study.

Table E-1Sensitivity analysis of MDCT vs. MRI for diagnosis

Which Readers were Used for the Four Studies Reporting Multiple Readers Separately	MDCT Sensitivity	MDCT Specificity	MRI Sensitivity	MRI Specificity
1,1,1,1 (primary analysis)	89% (95% CI: 82% to 94%)	90% (95% CI: 80% to 95%)	89% (95% CI: 81% to 94%)	89% (95% CI: 74% to 95%)
1,1,1,2	89% (95% CI: 83% to 94%)	90% (95% CI: 81% to 95%)	89% (95% CI: 80% to 94%)	88% (95% CI: 75% to 94%)
1,1,1,3	89% (95% CI: 83% to 94%)	90% (95% CI: 81% to 95%)	89% (95% CI: 81% to 94%)	89% (95% CI: 74% to 95%)
1,1,2,1	No convergence	No convergence	88% (95% CI: 80% to 93%)	90% (95% CI: 75% to 96%)
1,1,2,2	No convergence	No convergence	88% (95% CI: 80% to 93%)	89% (95% CI: 75% to 95%)
1,1,2,3	No convergence	No convergence	88% (95% CI: 80% to 93%)	90% (95% CI: 75% to 96%)
1,2,1,1	90% (95% CI: 82% to 94%)	90% (95% CI: 80% to 95%)	90% (95% CI: 80% to 95%)	89% (95% CI: 75% to 95%)
1,2,1,2	90% (95% CI: 83% to 94%)	90% (95% CI: 81% to 95%)	90% (95% CI: 80% to 95%)	88% (95% CI: 75% to 94%)
1,2,1,3	90% (95% CI: 83% to 94%)	90% (95% CI: 81% to 95%)	90% (95% CI: 80% to 95%)	89% (95% CI: 75% to 95%)
1,2,2,1	89% (95% CI: 82% to 93%)	90% (95% CI: 80% to 95%)	89% (95% CI: 80% to 94%)	90% (95% CI: 75% to 96%)
1,2,2,2	89% (95% CI: 82% to 93%)	90% (95% CI: 81% to 95%)	89% (95% CI: 79% to 94%)	89% (95% CI: 75% to 95%)
1,2,2,3	89% (95% CI: 82% to 93%)	90% (95% CI: 81% to 95%)	89% (95% CI: 80% to 94%)	90% (95% CI: 75% to 96%)
1,3,1,1	90% (95% CI: 82% to 94%)	90% (95% CI: 80% to 95%)	90% (95% CI: 80% to 95%)	89% (95% CI: 75% to 95%)
1,3,1,2	90% (95% CI: 83% to 94%)	90% (95% CI: 81% to 95%)	90% (95% CI: 80% to 95%)	88% (95% CI: 75% to 94%)
1,3,1,3	90% (95% CI: 83% to 94%)	90% (95% CI: 81% to 95%)	90% (95% CI: 80% to 95%)	89% (95% CI: 75% to 95%)
1,3,2,1	89% (95% CI: 82% to 93%)	90% (95% CI: 80% to 95%)	89% (95% CI: 80% to 94%)	90% (95% CI: 75% to 96%)
1,3,2,2	89% (95% CI: 82% to 93%)	90% (95% CI: 81% to 95%)	89% (95% CI: 79% to 94%)	89% (95% CI: 75% to 95%)
1,3,2,3	89% (95% CI: 82% to 93%)	90% (95% CI: 81% to 95%)	89% (95% CI: 80% to 94%)	90% (95% CI: 75% to 96%)
2,1,1,1	91% (95% CI: 84% to 95%)	89% (95% CI: 80% to 95%)	91% (95% CI: 82% to 95%)	89% (95% CI: 77% to 95%)
2,1,1,2	91% (95% CI: 84% to 95%)	90% (95% CI: 81% to 95%)	90% (95% CI: 81% to 95%)	88% (95% CI: 78% to 94%)
2,1,1,3	91% (95% CI: 84% to 95%)	90% (95% CI: 81% to 95%)	91% (95% CI: 82% to 95%)	89% (95% CI: 77% to 95%)
2,1,2,1	90% (95% CI: 84% to 94%)	89% (95% CI: 79% to 94%)	90% (95% CI: 81% to 94%)	90% (95% CI: 78% to 96%)
2,1,2,2	No convergence	No convergence	89% (95% CI: 81% to 94%)	89% (95% CI: 78% to 95%)
2,1,2,3	No convergence	No convergence	90% (95% CI: 81% to 94%)	90% (95% CI: 78% to 96%)
2,2,1,1	92% (95% CI: 84% to 96%)	89% (95% CI: 80% to 95%)	92% (95% CI: 82% to 96%)	89% (95% CI: 78% to 95%)
2,2,1,2	92% (95% CI: 84% to 96%)	90% (95% CI: 81% to 95%)	91% (95% CI: 81% to 96%)	88% (95% CI: 78% to 94%)
2,2,1,3	92% (95% CI: 84% to 96%)	90% (95% CI: 81% to 95%)	92% (95% CI: 82% to 96%)	89% (95% CI: 78% to 95%)
2,2,2,1	90% (95% CI: 84% to 94%)	89% (95% CI: 80% to 94%)	90% (95% CI: 81% to 95%)	90% (95% CI: 78% to 96%)
2,2,2,2	90% (95% CI: 84% to 94%)	90% (95% CI: 80% to 95%)	90% (95% CI: 81% to 95%)	89% (95% CI: 78% to 95%)
2,2,2,3	90% (95% CI: 84% to 94%)	90% (95% CI: 80% to 95%)	90% (95% CI: 81% to 95%)	90% (95% CI: 78% to 96%)
2,3,1,1	92% (95% CI: 84% to 96%)	89% (95% CI: 80% to 95%)	92% (95% CI: 82% to 96%)	89% (95% CI: 78% to 95%)
2,3,1,2	92% (95% CI: 84% to 96%)	90% (95% CI: 81% to 95%)	91% (95% CI: 81% to 96%)	88% (95% CI: 78% to 94%)
2,3,1,3	92% (95% CI: 84% to 96%)	90% (95% CI: 81% to 95%)	92% (95% CI: 82% to 96%)	89% (95% CI: 78% to 95%)
2,3,2,1	90% (95% CI: 84% to 94%)	89% (95% CI: 80% to 94%)	90% (95% CI: 81% to 95%)	90% (95% CI: 78% to 96%)
2,3,2,2	90% (95% CI: 84% to 94%)	90% (95% CI: 80% to 95%)	90% (95% CI: 81% to 95%)	89% (95% CI: 78% to 95%)
2,3,2,3	90% (95% CI: 84% to 94%)	90% (95% CI: 80% to 95%)	90% (95% CI: 81% to 95%)	90% (95% CI: 78% to 96%)

: Note: “No convergence” means that the metandi command in stata did not converge on estimates, even after increasing the number of integration points to the maximum of 15.

Table E-2Sensitivity analysis of MDCT vs. MRI for assessment of metastases

Which Reader was Used for the One Study Reporting Multiple Readers Separately	MDCT Sensitivity	MDCT Specificity	MRI Sensitivity	MRI Specificity
1 (primary analysis)	48% (95% CI: 31% to 66%)	90% (95% CI: 81% to 95%)	50% (95% CI: 19% to 81%)	95% (95% CI: 91% to 98%)
2	48% (95% CI: 31% to 65%)	91% (95% CI: 81% to 96%)	54% (95% CI: 18% to 86%)	96% (95% CI: 93% to 98%)

3	48% (95% CI: 31% to 65%)	91% (95% CI: 81% to 96%)	54% (95% CI: 18% to 86%)	96% (95% CI: 93% to 98%)

Table E-3Sensitivity analysis of MDCT vs. MRI for assessment of metastases

Which Reader was Used for the Two Study Reporting Multiple Readers Separately	MDCT Sensitivity	MDCT Specificity	MRI Sensitivity	MRI Specificity
1,1 (primary analysis)	48% (95% CI: 31% to 66%)	90% (95% CI: 81% to 95%)	50% (95% CI: 19% to 81%)	95% (95% CI: 91% to 98%)
2	48% (95% CI: 31% to 65%)	91% (95% CI: 81% to 96%)	54% (95% CI: 18% to 86%)	96% (95% CI: 93% to 98%)

3	48% (95% CI: 31% to 65%)	91% (95% CI: 81% to 96%)	54% (95% CI: 18% to 86%)	96% (95% CI: 93% to 98%)