Analyses and Risk of Bias Assessments

Jonathan Treadwell; Matthew Mitchell; Kelley Eatmon; Jane Jue; Hanna Zafar; Ursina Teitelbaum; Karen Schoelles

Appendix DAnalyses and Risk of Bias Assessments

Publication Details

Analyses of Comparative Accuracy

Table D-1

Summary of analyses of comparative accuracy.

Quality of Systematic Reviews

Modified AMSTAR Instrument¹³¹^,¹³² for Systematic Reviews

The eight items in boldface below were required to be answered “Yes” in order for a systematic review to be considered high quality. Otherwise, the review was rated not high quality.

1.: Was an a priori design or protocol provided?

2.: Was a comprehensive search strategy performed?

2a.: Was this strategy appropriate to address the relevant Key Question of the CER?

3.: Was a list of included and excluded studies provided?

4.: Was the application of inclusion/exclusion criteria unbiased?

4a.: Are the inclusion/exclusion criteria appropriate to address the relevant Key Question of the CER?

5.: Was there duplicate study selection and data extraction?

6.: Were the characteristics of the included studies provided?

7.: Was the individual study quality assessed?

7a.: Was the method of study quality assessment consistent with that recommended by the Methods Guide?

7b.: Was the scientific quality of the individual studies used appropriately in formulating conclusions?

8.: Were the methods used to combine the findings of studies appropriate?

9.: Was the likelihood of publication bias assessed?

10.: Have the authors disclosed conflicts of interest?

Table D-2

Quality assessments of systematic reviews.

Risk of Bias of Comparative Accuracy Studies

Did the study enroll all, consecutive, or a random sample of patients?
Was the study unaffected by spectrum bias (e.g., patients with known status before the study, or patients selected for being difficult to diagnose/stage)?
Was prior experience with the test (technicians, readers) similar for the two imaging tests being compared in the study?
Were the imaging tests performed within one month of each other (to avoid the possibility that the patient’s true condition changed between tests)?
Was knowledge of the other test complementary (either both tests were read with knowledge of the other results, or neither test was read with knowledge of the other)?
Did the interpreters have the same other information available at the time of interpretation for the two imaging tests (other clinical information, 3rd test results)?
Was each test’s accuracy measuring using the same reference standard (or a similar proportion of patients who underwent different reference standards such as clinical follow-up and surgical findings)?
Were readers of both tests of interest blinded to the results of the reference standard (or the reference standard was unknowable until after the tests were read)?
Were the people determining the reference standard unaware of the diagnostic test results?

We defined LOW risk of bias as a study that has a YES for the six boldfaced items above (#2, and #4-#8). We defined HIGH risk of bias as a study that has a NO (or Not Reported) for these six items. We defined MEDIUM risk of bias a study that meets neither the LOW nor the HIGH criteria.

Table D-3

Risk of bias assessments of comparative accuracy studies.

Publication Details

Copyright

Publisher

Agency for Healthcare Research and Quality (US), Rockville (MD)

NLM Citation

Treadwell J, Mitchell M, Eatmon K, et al. Imaging Tests for the Diagnosis and Staging of Pancreatic Adenocarcinoma [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014 Sep. (Comparative Effectiveness Review, No. 141.) Appendix D, Analyses and Risk of Bias Assessments.

Table D-1Summary of analyses of comparative accuracy

Comparison	Clinical Decision	# Studies	Measure	Test 1 Estimate and 95% CI^{^a}	Test 2 Estimate and 95% CI^{^a}	Logit Difference and 95% CI^b	Statistically Significantly Different?	Precise Enough to Indicate Approximately Equivalent Accuracy?
MDCT angiography without 3D reconstruction vs. with 3D reconstruction	Resectability in those not staged	1	Sensitivity	89% (95% CI: 68% to 97%)	100% (95% CI: 83% to 100%)	−1.5 (−4.3 to 1.2)	No	NA
MDCT angiography without 3D reconstruction vs. with 3D reconstruction	Resectability in those not staged	1	Specificity	79% (95% CI: 64% to 89%)	100% (95% CI: 91% to 100%)	−3 (−5.5 to −0.5)	Yes	See above cell
MDCT vs. EUS-FNA	Diagnosis	3	Sensitivity	87% (95% CI: 82% to 91%)	89% (95% CI: 85% to 93%)	−0.2 (−0.8 to 0.4)	No	No
MDCT vs. EUS-FNA	Diagnosis	3	Specificity	67% (95% CI: 53% to 78%)	81% (95% CI: 68% to 90%)	−0.7 (−1.7 to 0.2)	No	See above cell
MDCT vs. MRI	Diagnosis	7	Sensitivity	89% (95% CI: 82% to 94%)	89% (95% CI: 81% to 94%)	−0.01 (−1.4 to 1.5)	No	Yes
MDCT vs. MRI	Diagnosis	7	Specificity	90% (95% CI: 80% to 95%)	89% (95% CI: 74% to 95%)	0.1 (−2.5 to 2.8)	No	See above cell
MDCT vs. PET/CT	Diagnosis	6	Sensitivity	85% (95% CI: 80% to 90%)	91% (95% CI: 85% to 94%)	−0.6 (−1.2 to 0.1)	No	NA
MDCT vs. PET/CT	Diagnosis	6	Specificity	55% (95% CI: 44% to 66%)	72% (95% CI: 61% to 81%)	−0.7 (−1.4 to −0.1)	Yes	See above cell
EUS-FNA vs. PET/CT	Diagnosis	1	Sensitivity	81% (95% CI: 62% to 91%)	89% (95% CI: 72% to 96%)	−0.6 (−2.1 to 0.8)	No	No
EUS-FNA vs. PET/CT	Diagnosis	1	Specificity	84% (95% CI: 62% to 94%)	74% (95% CI: 51% to 88%)	0.6 (−0.9 to 2.2)	No	See above cell
MRI vs. PET/CT	Diagnosis	1	Sensitivity	85% (95% CI: 64% to 95%)	85% (95% CI: 64% to 95%)	0 (−1.6 to 1.6)	No	No
MRI vs. PET/CT	Diagnosis	1	Specificity	72% (95% CI: 49% to 87%)	94% (95% CI: 74% to 99%)	−1.9 (−3.8 to 0.1)	No	See above cell
MDCT vs. EUS-FNA	Resectability in those not staged	1	Sensitivity	64% (95% CI: 46% to 79%)	68% (95% CI: 49% to 82%)	−0.2 (−1.2 to 0.9)	No	Yes
MDCT vs. EUS-FNA	Resectability in those not staged	1	Specificity	92% (95% CI: 75% to 98%)	88% (95% CI: 70% to 96%)	0.4 (−1.3 to 2.2)	No	See above cell
MDCT vs. MRI	Resectability in those not staged	2	Sensitivity	68% (95% CI: 47% to 85%)	52% (95% CI: 31% to 72%)	0.7 (−0.6 to 1.9)	No	No
MDCT vs. MRI	Resectability in those not staged	2	Specificity	89% (95% CI: 77% to 96%)	91% (95% CI: 80% to 97%)	−0.2 (−1.7 to 1.2)	No	See above cell
MDCT vs. EUS-FNA	T staging	1	T staging	Accurate T stage in 41% (95% CI: 20/49); overstaged T in 14% (95% CI: 7/49), understaged T in 44% (95% CI: 22/49)	Accurate T stage in 67% (95% CI: 33/49); overstaged T in 18% (95% CI: 9/49), understaged T in 14% (95% CI: 7/49)	RR 0.61 (0.41 to 0.90)	Yes	NA
MDCT vs. EUS-FNA	Vessel involvement	1	Sensitivity	56% (95% CI: 34% to 75%)	61% (95% CI: 39% to 80%)	−0.2 (−1.5 to 1)	No	No
MDCT vs. EUS-FNA	Vessel involvement	1	Specificity	94% (95% CI: 80% to 98%)	91% (95% CI: 76% to 97%)	0.4 (−1.3 to 2.1)	No	See above cell
MDCT vs. MRI	T staging	1	T staging	Accurate T stage in 73% (95% CI: CI 62% to 84%), overstaging in 2% (95% CI: CI 0%–6%), and understaging in 25% (95% CI: CI 14%–36%).	Accurate T stage in 62% (95% CI: CI 49% to 75%), overstaging in 6% (95% CI: CI 0%–12%), and understaging in 32% (95% CI: CI 19%–45%).	RR 1.17 (0.90 to 1.52)	No	No
MDCT vs. MRI	N staging	1	Sensitivity	38% (95% CI: 21% to 57%)	15% (95% CI: 5% to 36%)	1.2 (−0.2 to 2.6)	No	No
MDCT vs. MRI	N staging	1	Specificity	79% (95% CI: 63% to 90%)	93% (95% CI: 78% to 98%)	−1.3 (−2.8 to 0.2)	No	See above cell
MDCT vs. MRI	Metastases	5	Sensitivity	48% (95% CI: 31% to 66%)	50% (95% CI: 19% to 82%)	−0.09 (−1.2 to 1.0)	No	No
MDCT vs. MRI	Metastases	5	Specificity	90% (95% CI: 81% to 95%)	95% (95% CI: 91% to 98%)	−0.9 (−2.2 to 0.9)	No	See above cell
MDCT vs. MRI	Precise stage	1	Precise stage	Accurate TNM stage in 46% (95% CI: CI 33% to 59%), overstaging in 8% (95% CI: CI 1%–15%), and understaging in 46% (95% CI: CI 33%–59%).	Accurate TNM stage in 36% (95% CI: CI 23% to 49%), overstaging in 7% (95% CI: CI 0%–14%), and understaging in 57% (95% CI: CI 44%–70%).	RR 1.28 (0.81 to 2.01)	No	No
MDCT vs. MRI	Vessel involvement	2	Sensitivity	68% (95% CI: 55% to 79%)	62% (95% CI: 48% to 74%)	0.3 (−0.5 to 1.1)	No	Yes
MDCT vs. MRI	Vessel involvement	2	Specificity	97% (95% CI: 94% to 98%)	96% (95% CI: 93% to 98%)	0.3 (−0.6 to 1.2)	No	See above cell
MDCT vs. MRI	Resectability in those staged	1	Sensitivity	67% (95% CI: 48% to 81%)	57% (95% CI: 37% to 74%)	0.4 (−0.7 to 1.5)	No	No
MDCT vs. MRI	Resectability in those staged	1	Specificity	97% (95% CI: 84% to 99%)	90% (95% CI: 74% to 96%)	1.2 (−0.8 to 3.2)	No	See above cell
MDCT vs. PET/CT	N staging	1	Sensitivity	26% (95% CI: 14% to 43%)	32% (95% CI: 19% to 50%)	−0.3 (−1.4 to 0.8)	No	Yes
MDCT vs. PET/CT	N staging	1	Specificity	75% (95% CI: 50% to 90%)	75% (95% CI: 50% to 90%)	0 (−1.5 to 1.5)	No	See above cell
MDCT vs. PET/CT	Metastases	2	Sensitivity	57% (95% CI: 37% to 75%)	67% (95% CI: 47% to 83%)	−0.4 (−1.6 to 0.8)	No	NA
MDCT vs. PET/CT	Metastases	2	Specificity	91% (95% CI: 81% to 97%)	100% (95% CI: 95% to 100%)	−2.3 (−4.5 to −0.1)	Yes	See above cell
EUS-FNA vs. MRI	Precise stage	1	Precise stage	Accurate stage for 34/48 patients who had undergone surgical exploration. Of the 34, 34 were stage 2 and below, and 0 was stage 3 or above. The test understaged 13/48, and overstaged 1/48.	Accurate stage for 36/48 patients who had undergone surgical exploration. Of the 36, 35 were stage 2 and below, and 1 was stage 3 or above. The test understaged 12/48, and overstaged 0/48.	RR 0.94 (0.74 to 1.21)	No	Yes
MRI vs. PET/CT	Metastases	1	Sensitivity	57% (95% CI: 25% to 84%)	86% (95% CI: 48% to 97%)	−1.5 (−3.7 to 0.7)	No	No
MRI vs. PET/CT	Metastases	1	Specificity	86% (95% CI: 48% to 97%)	94% (95% CI: 64% to 100%)	−0.9 (−4 to 2.2)	No	See above cell

a: If multiple studies, this is the random-effects summary estimate, but if only one study, this is the single-study estimate

b: For most rows, this column indicates the results of statistical comparison of the two tests using equation 39 of Trikalinos.¹⁷ A positive logit difference favors test 1, and a negative logit difference favors test 2. For rows with RR (relative risk), it is the results of the statistical comparison of the two rates using relative risk; RR>1 favors test 1 and RR<1 favors test 2.

: NA=Not applicable since the question of equivalence does not apply when a statistically significant difference exists for either sensitivity or specificity; RR=relative risk

Table D-2Quality assessments of systematic reviews

Study	1	2	2a	3	4	4a	5 Sel.	5 Ext.	6	7	7a	7b	8	9	10	Meets Eight Most Important Criteria (High Quality)
Affolter et al. 2013¹	No	Yes	No	No	No	No	Yes	Yes	Yes	No	No	No	Yes	No	Yes	No
Chen et a. 2013²	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes
Hébert-Magee et al. 2013³	No	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	No	No
Li et al. 2013¹⁴	No	No	Yes	No	Yes	Yes	No	Yes	Yes	No	No	No	Yes	No	Yes	No
Madhoun et al. 2013⁴	No	Yes	Yes	No	Yes	Yes	Yes	No	No	Yes	Yes	No	Yes	No	Yes	Yes
Wang et al. 2013⁵	No	Yes	Yes	No	Yes	Yes	No	Yes	Yes	No	No	No	Yes	Yes	No	No
Puli et al. 2013⁶	No	Yes	Yes	No	Yes	Yes	Yes	Yes	No	Yes	Yes	No	Yes	Yes	No	No
Chen et al. 2012⁷	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes	No	Yes	Yes
Hewitt et al. 2012⁸	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes
Wu et al. 2012⁹	No	No	No	No	Yes	Yes	No	Yes	Yes	No	No	No	Yes	No	Yes	No
Wu et al. 2012¹⁰	No	No	No	No	Yes	Yes	Yes	Yes	No	No	No	No	No	No	Yes	No
Tang et al. 2009¹¹	No	No	No	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	Yes	No	Yes	No
Zhao et al. 2009¹⁵	No	No	No	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	No	Yes	No	Yes	No
Hartwig et al. 2008¹²	No	Yes	Yes	No	Yes	Yes	No	Yes	No	Yes	Yes	No	Yes	Yes	Yes	Yes
Bipat et al. 2005¹³	No	No	No	No	Yes	Yes	Yes	No	Yes	Yes	Yes	No	Yes	Yes	Yes	No

: Quality criteria: 1–A priori design or protocol provided, 2–Comprehensive search performed, 2a–Search strategy appropriate to address key questions of this review, 3–Lists of both included and excluded studies provided, 4–Inclusion/exclusion criteria applied in an unbiased manner, 4a–Inclusion/exclusion criteria appropriate to address key questions of this review, 5 sel.–Study selection done in duplicate, 5 ext.–Data abstraction done in duplicate, 6–Characteristics of individual studies reported in evidence table, 7–Quality of each individual study assessed, 7a–Quality assessment consistent with that recommended by the Methods Guide, 7b–Quality of the individual studies used appropriately in formulating conclusions, 8–Data synthesis methods appropriate, 9–The likelihood of publication bias assessed in a qualitative or quantitative manner, 10–Conflicts of interest disclosed by authors. Questions 2, 2a, 4, 4a, 7, 7a, 8, and 10 were deemed “most important.”

Table D-3Risk of bias assessments of comparative accuracy studies

Study	1	2	3	4	5	6	7	8	9	Comments	Risk of Bias
Fang et al. 2012¹⁶	NR	Y	NR	Y	Y	Y	Y	Y	Y	–	Low
Herrmann et al. 2012¹⁷	NR	Y	Y	N	Y	Y	Y	Y	NR	–	Moderate
Tellez-Avila et al. 2012¹⁸	NR	Y	Y	NR	NR	NR	Y	Y	Y	Review of data obtained prospectively - EUS and CT - of pancreas lesion that then went to OR for surgical resection with goal of study being detection of vascular invasion, i.e., status of resectability. Not clearly stated but probably results of all tests to date available to all readers.	Moderate
Holzapfel et al. 2011¹⁹	Y	Y	NR	Y	Y	Y	Y	Y	NR	Two radiologists looked at each images together and came to consensus. Analysis of MDCT and MRI images were spaced 4 weeks apart to avoid any learning bias	Low
Koelblinger et al. 2011²⁰	Y	Y	Y	Y	Y	Y	Y	Y	NR	Reading sessions for CT and MR were separated by at least 8 weeks to minimize recall bias, and images were presented to readers in a different randomized order	Low
Motosugi et al. 2011²¹	NR	Y	N	Y	Y	Y	Y	Y	Y	–	Low
Rao et al. 2011²²	NR	N	Y	NR	Y	Y	Y	Y	NR	Bias against MRI because patients only had MRI if their case was more difficult (see Discussion section of the article). Small tumors only, which are harder to detect, thus possible spectrum bias.	Moderate
Shami et al. 2011²³	NR	Y	NR	NR	NR	NR	Y	Y	NR	Radiologists for MRI were blinded to EUS result, but did not report the order of the tests or whether EUS readers were blind to MRI result	Moderate
Takakura et al. 2011²⁴	NR	Y	Y	N	Y	Y	Y	Y	Y	Has flowchart for included patients, but doesn’t say consecutive or all	Low
Imai et al. 2010²⁵	NR	N	NR	NR	Y	Y	Y	Y	NR	Possible spectrum bias because authors imaged for the presence of a particular kind of mets that is hard to detect (para-aortic lymph node metastasis or PALN)	Moderate
Lee et al. 2010²⁶	Y	Y	Y	Y	Y	Y	Y	Y	NR	The interval between reads was 2 weeks to minimize learning bias.	Low
Kauhanen et al. 2009²⁷	Y	Y	NR	Y	Y	Y	Y	Y	NR	–	Low
Farma et al. 2008²⁸	NR	Y	NR	NR	NR	NR	Y	NR	Y	All patients had a peroperative biopsy performed by percutaneous or endoscopi means. Clinical, radiographic, and pathologic follw-up was evaluated for each patient	Moderate
Saif et al. 2008²⁹	NR	Y	NR	Y	Y	Y	Y	Y	NR	–	Low
Schick et al. 2008³⁰	Y	Y	NR	Y	NR	NR	Y	Y	NR	–	Moderate
Casneuf et al. 2007³¹	Y	Y	NR	NR	N	Y	Y	Y	Y	One reader for MDCT, and two readers together for PET/CT (one of whom had read the MDCT image at least 2 months earlier, and one who had read the PET alone image 2 at least months earlier). For PET/CT they had to come to consensus. This design is a bias in favor of PET/CT because there were always 4 eyes on the PET/CT, whereas CT only had 2 eyes. In addition, the PET/CT assessment was probably influenced (improved?) by the two readers’ prior memory of the two individual scans.	Moderate
Tamm et al. 2007³²	NR	Y	NR	NR	N	N	Y	Y	N	MDCT images were read without knowledge of clinical, pathologic, or surgical data, or EUS-FNA findings. EUS-FNA was performed with knowledge of the MDCT finding. Also the reference standard for some patients was determined by the FNA. This design is a bias in favor of EUS-FNA.	Moderate
Mehmet Ertuk et al. 2006³³	Y	N	NR	Y	Y	Y	Y	Y	Y	Patient statuses were all known beforehand, hence probably spectum bias. MDCT was always read first. The interval between reads was 4 weeks to minimize learning bias.	Moderate
Heinrich et al. 2005³⁴	NR	Y	NR	Y	NR	NR	Y	NR	NR	Findings on PET/CT were compared with results of standard staging and validated by intraoperative findings and histology of the resected specimen or biopsies. For patients who were diagnosed to have benign pancreatic lesion by PET/CT and did not undergo resection, long-term outcome was assessed to confirm the diagnosis made by PET/CT	Moderate
Agarwal et al. 2004³⁵	Y	Y	NR	NR	NR	Y	Y	Y	Y	–	Moderate
DeWitt et al. 2004³⁶	NR	Y	Y	Y	Y	Y	Y	Y	N	Readers for neither test were blind to previous radiographic data. Readers of the 2nd test (which was always MDCT) were blind to results from the 1st (which was always EUS-FNA)	Low
Lemke et al. 2004³⁷	NR	Y	NR	Y	N	NR	Y	NR	NR	2 radiologists evaluated the original CT and PET images as well as the fused images in a randomized order in 3 different settings with an interval of 2 weeks each, using a standardized questionnaire.	Moderate
Soriano et al. 2004³⁸	Y	Y	NR	NR	Y	Y	Y	Y	Y	Surgeons only saw a combined report of all the imaging tests, and did not know individual imaging results	Moderate
Rieber et al. 2000³⁹	NR	Y	NR	NR	N	Y	Y	Y	NR	Readers: 3 different radiologists blinded to all clinical data regarding the patient.	Moderate

Appendix DAnalyses and Risk of Bias Assessments

Analyses of Comparative Accuracy

Table D-1

Quality of Systematic Reviews

Modified AMSTAR Instrument131,132 for Systematic Reviews

Table D-2

Risk of Bias of Comparative Accuracy Studies

Table D-3

Publication Details

Copyright

Publisher

NLM Citation

Table D-1Summary of analyses of comparative accuracy

Table D-2Quality assessments of systematic reviews

Table D-3Risk of bias assessments of comparative accuracy studies

Modified AMSTAR Instrument¹³¹^,¹³² for Systematic Reviews