Appendix ERisk of Bias Assessment

Additional Comments on Studies Rated High Risk of Bias

Ahmadizadeh et al., 2013:¹⁹⁸ No details about randomization, blinding, completion of treatment, fidelity, how dropouts were handled in analyses.

Arntz et al., 2007:¹⁹⁹ Very high attrition and high differential attrition (just 45% completed in one group, 72% in the other); outcome assessor and randomization procedures unclear; outcome assessors not described as masked; no description of treatment fidelity.

Beck et al., 2009:²⁰⁰ High risk of attrition bias, due to the overall and the differential attrition (24% difference between groups). Unclear whether groups were similar at baseline for demographics and most potential confounders (as the information is not provided). In addition, inadequate handling of missing data (used completers analysis). No description of randomization method or allocation concealment.

Beidel et al., 2011:²⁰¹ High risk of selection bias; completers analysis in a small trial (N=35) with high differential dropout; and risk of bias from no masking.

Bichescu et al., 2007:²⁰² No attempt to create similar groups, this subsequently affected assessor blinding. Few details of randomization process beyond “were randomized”.

Braun et al., 1990:¹⁶⁹ High attrition, non-standard outcome measures, baseline data not reported to allow determination of similarity or differences between groups.

Brom et al., 1989:²⁰³ Appears to be completers analysis with no approach to handling missing data reported; no data reported to allow comparison of groups at baseline; no masking of outcome assessors reported; no information on treatment fidelity; methods of randomization and allocation concealment not reported; potential measurement bias due to differences in timing of assessments across groups.

Butollo et al., 2016:²⁰⁴ Very large loss to followup, do not know details about randomization or allocation concealment or outcome assessor masking, fidelity reported as tested but details not provided on adequacy. Self-report primary outcomes.

Davidson et al., 1993:¹⁸⁰ Davidson, 1990:¹⁷⁹ Completer analysis for all subjects completing minimum of 4 weeks (40/46 subjects did so and were included in the analyses, 87%); and separately for the 33/46 (71.1%) that completed 8 weeks; no treatment of missing data; with high attrition. It was also unclear whether randomization or allocation concealment were adequate.

Davis et al., 2004:²⁰⁵ Very high attrition (close to 50% overall); groups mostly similar at baseline but differed in prior treatments (with just 1 subject in the placebo group previously treated with an antidepressant vs. 15 to 27% of subjects with previous treatment with antidepressants, benzodiazepines, or other medication in the nefazodone group); ITT analysis with LOCF (with exception of 1 patient).

Difede et al., 2007:²⁰⁶ Very high attrition, and high differential attrition; over 1/2 for the CBT group did not complete treatment.

Dorrepaal et al., 2012:²⁰⁷ Groups differ by those with adult abuse and trauma severity scores, and analyses did not adjust for these baseline differences. Outcome assessors of primary outcomes not blinded because they are self-reported. Only secondary outcome assessors were blinded. Randomization methods not reported, no fidelity details given. Providers and patients not masked.

Dunne et al., 2012:²⁰⁸ No randomization or allocation concealment details, text states that outcome assessors were not blinded.

Feske et al., 2008:²⁰⁹ High risk of selection bias and confounding; already small sample size and the high overall and differential attrition with completer analysis; attrition bias; 4 of 13 randomized subjects in the prolonged exposure group (31%) dropped out, 2 were withdrawn due to medication changes and 2 for unknown reasons; 2/14 treatment as usual clients withdrawn.

Foa et al., 1991:²¹⁰ High attrition for some groups and high differential attrition; completer analysis only; study did not report adequate treatment fidelity; some baseline differences between groups for income, assault characteristics; high risk of selection bias and confounding.

Franklin et al., 2017:²¹¹ Could not determine whether groups were similar at baseline, large loss to followup, completer analysis.

Frommberger et al., 2004:²¹² High risk of selection bias and confounding; attrition bias; no reporting of adequate fidelity; Small sample size with no data shown on baseline covariates across groups; outcome assessment not masked; over 20% attrition and nothing done for missing data (completer analysis).

Galovski et al., 2016:²¹³ Unclear whether patients and providers were blinded. High overall attrition.

Ghafoori et al., 2017:²¹⁴ Differences in race at baseline, but probably ok; large loss to followup, outcome assessors not blind to treatment allocation.

Hamner et al., 2009:²¹⁵ Substantial dropout, limited description of randomization; study reported as double blind, but write up suggests VPA folks got a lot more blood draws/monitoring; also, study physician told by pharmacist to adjust doses, so not blind to treatment arm.

Hensel-Dittman et al., 2011:²¹⁶ High risk of selection bias and confounding. First, no data were reported to allow baseline comparison of groups for most variables, and this is a fairly small sample size, making baseline differences more likely. The authors only report baseline data for a few of the outcome measures, and there was an 11-point difference between groups for baseline CAPS score. They did some matching during the randomization, but it is unclear if that worked to produce comparable groups at baseline. Next, the study did not report adequate treatment fidelity based on measurement by independent raters; no information was reported about treatment fidelity. They report that they videotaped all sessions, but there is no information reported to confirm to support adequate treatment fidelity, which would be very important since all of the same therapists delivered both interventions and it would be fairly easy to have some of the components of one therapy introduced into the other therapy. Next, lack of masking; the authors report that they attempted to keep outcome assessors blind, but that treatment condition was occasionally revealed to them, but it is unclear how frequently this occurred.

Hertzberg et al., 1999:²¹⁷ Baseline characteristics not reported for important potential confounders in this small study (n =15) to allow for determination of potential selection bias; in addition, unclear whether randomization or allocation concealment were adequate; unclear whether outcome assessors were masked. Completers analysis.

Hertzberg et al., 2000:¹⁷⁸ Baseline characteristics not reported for important potential confounders in this small study (n=12) to allow for determination of potential selection bias (described as “non-significant difference”, but given small sample size, almost any difference will be non-significant). In addition, unclear whether randomization or allocation concealment were adequate; unclear whether outcome assessors were masked. Instruments of uncertain validity used to assess outcomes.

Ironson et al., 2002:²¹⁹ High risk of selection bias; randomization compromised by adding more participants to PE group to achieve equal group numbers; high overall and differential attrition (and 50% dropouts from the PE group); marked differences in baseline severity of PTSD and depression between groups (otherwise, minimal baseline data reported to allow comparison of groups); completer analysis; no handling of missing data.

Jiang et al., 2014:²⁶⁰ Very high attrition and differential attrition. Small study so no one could be masked, main outcome measures not validated in Chinese population. Fidelity reported as assessed but findings not reported.

Johnson et al., 2006:²²¹ Inadequate methods of handling missing data, completers analysis; did not report adequate treatment fidelity based on measurement by independent raters; high potential for selection bias with small numbers in each treatment arm and no reporting of baseline demographics (only reported in aggregate for the three intervention groups) and potential confounders for comparison, and there were differences in the baseline values for the measures of PTSD symptoms (e.g., baseline CAPS scores were 82 for Counting and 61.7 for EMDR, 64.2 for waitlist). The authors describe the study as a randomized trial. However, from their description of the design, it appears that the participants for the waitlist control group were recruited separately from the group recruited to the active treatments. In other words, participants recruited to the active condition were randomized to one of three active treatments, but the persons recruited to the control condition were not assigned to that group randomly. Accordingly, it’s not really a randomized trial for the comparisons with the control condition.

Karatzias et al., 2011:²²² Very high attrition rate (over 40%); unclear whether randomization or allocation concealment were adequate.

Keane et al., 1989:²²³ High risk of selection bias: Baseline differences between groups included race (for Intervention vs. waitlist: 0% vs. 31% Black), and service connection (36% vs. 69%) possibly biasing control group toward reporting greater severity of symptoms; difference between group in co-interventions/medications administered over the course of the study (42.9% [6/14] in intervention group received anxiolytic, sleep, or pain meds at some point during the study vs. 76.9% [10/13] in the control group received anxiolytic medications at some point during waiting; and some evidence suggests worse outcomes for those with PTSD treated with anxiolytics). The PTSD ratings were completed by therapists who were administering the therapy and thus were not blinded. Of note, the study found no difference between active intervention and control group in self-reported PTSD symptoms but a substantial difference in PTSD ratings completed by the non-blinded therapists. Potential measurement bias with no masking or independence of outcome assessors and outcomes assessed at different timepoints for the two groups. Unclear whether randomization, allocation concealment, and masking were adequate. Attrition information not reported, nor was approach to handling missing data. No description of methods to ensure treatment fidelity.

Knaevelsrud et al., 2015:²²⁴ No blinding, very large loss to followup, no adherence assessed.

Krakow et al., 2000:²²⁵ Very high attrition, around 50%; did not report adequate treatment fidelity.

Krupnick et al., 2008:²²⁶ High risk of selection bias due to attrition. Very high attrition and high differential attrition (% completers by group: 63 vs. 44). Regarding “other” method of handling dropouts: imputed missing scores as the application of the observed group mean change.

Lee et al., 2002:²²⁷ Inadequate randomization procedure (alternating); no allocation concealment, no blinding of outcome assessors; unclear whether groups were similar at baseline for several characteristics; details of analysis and missing data were NR; differential attrition data unclear.

Lindley et al., 2007:²²⁸ High attrition and high differential attrition (30%), method of handling dropouts/missing data was unclear.

Littleton et al., 2016:²²⁹ Very high attrition, allocation concealment unclear, masking unclear of outcome assessors.

Marcus et al., 1997:²³⁰ No data reported to allow assessment of how groups compare at baseline, how many patients dropped out after randomization, or how many people are in the 2 groups. Attrition information not reported; does not describe use of ITT analysis; Outcome assessors were not masked, increasing potential for measurement bias; did not report adequate treatment fidelity.

Margolies et al., 2013:²³¹ Unclear randomization, allocation concealment, and all blinding, large overall attrition, fidelity not assessed.

Marshall et al., 2007:¹⁷⁷ High risk of selection bias due to high rate of attrition. Also, not clear if groups were similar at baseline (article does not show the data--it just has a sentence that says that patient demographics did not differ significantly between groups; although later Tables do show similar baseline PTSD severity for CAPS and some other measures).

McLay et al., 2011:²³² Unclear adequacy of randomization or allocation concealment; unclear whether or not outcome assessors were masked; small sample with possible significant differences in prior deployments between treatment groups, raising risk of selection bias. The measures themselves were reliable but post assessments were reported to be given sporadically over a 36-week period. Study did not report adequate treatment fidelity.

McRae et al., 2004:²³³ Completers analysis with inadequate handling of missing data in this head-to-head study that found no difference between treatments; high risk of selection bias; unable to determine if randomized groups were similar at baseline (data only reported for completers; 26/37 subjects); unclear whether randomization and allocation concealment were adequate.

Mueser et al., 2015:²³⁴ High attrition and, for some time points, differential attrition. Reported percent completion at 6 months because that was the lowest percent assessed in each group but did not account for treatment engagement.

Naylor et al., 2015:²³⁵ No randomization details presented, high attrition and differential attrition, baseline differences not accounted for in analyses, small sample sizes.

Niles et al., 2012:²³⁶ Providers and patients only masked until the end of the first assessment; Therapist compliance is mentioned but details are not presented; Unclear about outcome assessor blinding; Did not account for dropouts in analysis.

Noohi et al., 2017:²³⁷ This is a very small study (n-=30) with virtually no information reported to allow us to assess risk of bias. The authors say the participants were randomized but no other information was provided; no masking information was reported, no loss to followup data, no information about how missing data were handled, etc.

Padala et al., 2006:²³⁸ High risk of selection bias and confounding; differential attrition along with small sample size (N=20); completer analysis; only reports age, race, mean TOP-8, and mean CAPS at baseline---the race characteristics were quite different (55% Caucasian in Risperidone group vs. 89% in the Placebo group).

Paunovic et al., 2001:²³⁹ High risk of selection bias and confounding; high differential attrition in this small (N=20) head to head study comparing two types of psychotherapy that found no difference between the two, and was not powered to find a small to moderate difference between treatments; no assessor masking; did not reported whether ITT; handling of missing data not reported.

Petrakis et al., 2016:²⁴¹ Randomization methods/allocation concealment/masking are all unclear in the text; authors state it is a double-blind study but do not provide details. Only 56% of patients remained on study medication for 12 weeks (40% in G1, 48% in G2). Authors do not explain how dropouts/missing data are handled in analyses. Consort table reports that 100% of randomized patients received allocated intervention but results text suggests that 1 patient randomized to placebo reported wrong medication being dispensed. Additionally, 1 patient randomized to placebo reported that the “medication blind” envelope was not properly filed. Poor quality is due to high attrition and low treatment completion rate as well as unclear randomization/concealment/masking methods. Of note is that results may not be generalizable to outpatient settings since a majority of the participants were in a residential substance use program.

Popiel et al., 2015:²⁴² High attrition, nearly 50% of paroxetine group refused to participate in assigned treatment.

Power et al., 2002:²⁴³ High overall and differential attrition; completers analysis; no approach to handling missing data; no assessment of treatment fidelity; in the two active treatment groups, about 31% and 43% did not complete treatment, respectively.

Rauch et al., 2009:²⁴⁴ High risk of selection bias and confounding; completers analysis, using just the set of subjects that completed an RCT (Foa et al 2005, J Consul Clin Psychol); baseline differences in race and income.

Rauch et al., 2015:²⁴⁵ No randomization or allocation concealment details reported, patients and providers not masked, large differential attrition, no information on baseline group differences, no accounting for those lost to followup, no fidelity assessment.

Ready et al., 2010:²⁴⁶ High risk of selection bias and confounding. This small study (N = 11) did not report differences in many baseline covariates across intervention groups. However, there were large differences in some of the few that they did report (CAPS, BDI), which strongly suggests that there were important differences in baseline covariates.

Reist et al., 1989:¹⁸¹ Non-standard outcome measures, high attrition, only overall attrition not group-specific attrition reported, completer analysis.

Rosaura Polak, 2015:²⁴⁷ Very small n=8 pilot study with unclear randomization, allocation concealment, assessor blinding, and fidelity.

Rothbaum et al., 2008:²⁴⁸ Randomization unclear, high differential attrition (36% differential), completer’s analysis; unclear whether outcome assessor were masked.

Schneier et al., 2015:²⁴⁹ Very high attrition, randomization unclear.

Simpson, 2015:²⁵⁰ Attrition reported for original 12 week treatment period. During the course of the study, the investigators decided to stop the study at 6 weeks because they were concerned about the study length. Randomization also unclear.

Stecker, 2014:²⁵¹ Unknown whether allocation concealment or blinding of outcome assessors occurred; no fidelity assessed; large loss to followup; analyses not done on ITT basis and did not account for attrition; no provider and patient masking; and all information collected was self-report.

Stenmark, 2013:²⁵² Randomization method was drawing balls from a bag (presumably no allocation concealment). Baseline characteristics similar for gender, age, months in exile and region of origin; no characteristics reported for comorbidity, length of PTSD or other clinical factors. Authors used therapists from other centers to assess diagnostic status and symptom severity; attempts were made at blinding, but at least 20% of patients revealed treatment information to assessors. Percent completed treatment refers to those who completed 1 month and 6-month post-treatment testing and was high; authors report ITT analyses but exact method for addressing missing outcome data unclear.

Suris, 2013:²⁵³ A therapist had really poor fidelity so the authors removed all participants counseled by that therapist, resulting in very high loss to followup of randomized sample. Even so, the analyzed sample still had substantial loss to followup and differential loss to followup. Allocation concealment not clear and no blinding of patients or providers.

Ulmer et al., 2011:²⁵⁴ High risk of selection bias and confounding in this small study (N=22); differential attrition (% completers: 82 vs. 67 vs. 100); no description of treatment fidelity; unclear adequacy of randomization and allocation concealment; no masking of outcome assessors. Also, participants received a range of treatments outside of the study varying in intensity and type.

Vera et al., 2011:²⁵⁵ Very small (n=14) study. No information about randomization, allocation concealment, or masking of outcome assessors or patients. Differential attrition. Baseline CAPS scores significantly higher for usual care group than PE group. No ITT analysis done. No treatment fidelity reported (although therapists were supervised).

Villarreal et al., 2016:²⁵⁶ Large attrition and differential attrition. Baseline differences between groups (although authors adjusted for those variables in analyses). Also, the paper says it is a double-blind trial but authors do not describe who was masked to allocation.

Wagner et al., 2007:²⁵⁷ High risk of selection bias and confounding in this small study (N=8) with randomization method unclear, and groups different at baseline (younger in treatment group: mean age 28 vs. 39; more males 75% vs. 0%; more prior trauma and greater injury severity); no description of treatment fidelity; single therapist.

Wahbeh et al., 2016:²⁵⁸ Loss to followup not reported; outcome assessors not blinded; completer analysis only; no fidelity measure.

Zlotnick et al., 1997:²⁵⁹ High attrition (31%) with completers analysis; no masking of outcome assessors; baseline data not reported to allow comparison of groups for many things (they did run statistical tests for some demographic variables, and report no statistically significant differences); higher baseline scores for DTS, CR-PTSD, and DES for the wait list group.