Appendix FQuality Assessment Methods

For Controlled Trials

Each criterion was give an assessment of yes, no, or unclear.

1. Was the assignment to the treatment groups really random?
   • Adequate approaches to sequence generation:
     • Computer-generated random numbers
     • Random numbers tables
   • Inferior approaches to sequence generation:
     • Use of alternation, case record numbers, birth dates or week days
   • Randomization reported, but method not stated
   • Not clear or not reported
   • Not randomized
2. Was the treatment allocation concealed?
   • Adequate approaches to concealment of randomization:
     • Centralized or pharmacy-controlled randomization (randomization performed without knowledge of patient characteristics).
     • Serially-numbered identical containers
     • On-site computer based system with a randomization sequence that is not readable until allocation
     • Sealed opaque envelopes
   • Inferior approaches to concealment of randomization:
     • Use of alternation, case record numbers, birth dates or week days
     • Open random numbers lists
     • Serially numbered non-opaque envelopes
     • Not clear or not reported
3. Were the groups similar at baseline in terms of prognostic factors?
4. Were the eligibility criteria specified?
5. Were outcome assessors and/or data analysts blinded to the treatment allocation?
6. Was the care provider blinded?
7. Was the patient kept unaware of the treatment received?
8. Did the article include an intention-to-treat analysis, or provide the data needed to calculate it (i.e., number assigned to each group, number of subjects who finished in each group, and their results)?
9. Did the study maintain comparable groups?
10. Did the article report attrition, crossovers, adherence, and contamination?
11. Is there important differential loss to followup or overall high loss to followup?

For Cohort Studies

Each criterion was give an assessment of yes, no, or unclear.

1. Did the study attempt to enroll all (or a random sample of) patients meeting inclusion criteria, or a random sample (inception cohort)?
2. Were the groups comparable at baseline on key prognostic factors (e.g., by restriction or matching)?
3. Did the study use accurate methods for ascertaining exposures, potential confounders, and outcomes?
4. Were outcome assessors and/or data analysts blinded to treatment?
5. Did the article report attrition?
6. Did the study perform appropriate statistical analyses on potential confounders?
7. Is there important differential loss to followup or overall high loss to followup?
8. Were outcomes prespecified and defined, and ascertained using accurate methods?

For Case-Control Studies

Each criterion was given an assessment of yes, no, or unclear.

1. Did the study attempt to enroll all (or a random sample of) cases using predefined criteria?
2. Were the controls derived from the same population as the cases, and would they have been selected as cases if the outcome was present?
3. Were the groups comparable at baseline on key prognostic factors (e.g., by restriction or matching)?
4. Did the study report the proportion of cases and controls who met inclusion criteria that were analyzed?
5. Did the study use accurate methods for identifying outcomes?
6. Did the study use accurate methods for ascertaining exposures and potential confounders?
7. Did the study perform appropriate statistical analyses on potential confounders?

Systematic Reviews

Each criterion was given an assessment of yes, no, unclear, or not applicable.

1. Was an “a priori” design provided?
   The research question and inclusion criteria should be established before the conduct of the review.
2. Was there duplicate study selection and data extraction?
   There should be at least two independent data extractors and a consensus procedure for disagreements should be in place.
3. Was a comprehensive literature search performed?
   At least two electronic sources should be searched. The report must include years and databases used (e.g. Central, Embase, and MEDLINE). Key words and/or MeSH terms must be stated and where feasible the search strategy should be provided. All searches should be supplemented by consulting current contents, reviews, textbooks, specialized registers, or experts in the particular field of study, and by reviewing the references in the studies found.
4. Was the status of publication (i.e., gray literature) used as an inclusion criterion?
   The authors should state that they searched for reports regardless of their publication type. The authors should state whether or not they excluded any reports (from the systematic review), based on their publication status, language, etc.
5. Was a list of studies (included and excluded) provided?
   A list of included and excluded studies should be provided.
6. Were the characteristics of the included studies provided?
   In an aggregated from such as a table, data from the original studies should be provided on the participants, interventions and outcomes. The ranges of characteristics in the studies analyzed, e.g., age, race, sex, relevant socioeconomic data, disease status, duration, severity, or other diseases should be reported.
7. Was the scientific quality of the included studies assessed and documented?
   ‘A priori’ methods of assessment should be provided (e.g., for effectiveness studies if the author(s) chose to include only randomized, double-blind, placebo controlled studies, or allocation concealment as inclusion criteria); for other types of studies alternative items will be relevant.
8. Was the scientific quality of the include studies used appropriately in formulating conclusions?
   The results of the methodological rigor and scientific quality should be considered in the analysis and the conclusions of the review, and explicitly stated in formulating the recommendations.
9. Were the methods used to combine the findings of studies appropriate?
   Reviews should not combine or pool dissimilar studies. If studies are pooled using a fixed effects model, there should be a clear rationale for doing so. A test should be done to assess for statistical heterogeneity (i.e., Chi-squared test for homogeneity, I²).
10. Was the likelihood of publication bias assessed?
    An assessment of publication bias should include a combination of graphical aids (e.g., funnel plot, other available tests) and/or statistical tests (e.g., Egger regression test). If assessment of publication bias is not possible, the review should provide justification (e.g., small numbers of studies, too much heterogeneity, poor quality, etc.)
11. Was the conflict of interest stated?
    Potential sources of support should be clearly acknowledged in both the systematic review and the included studies.

Footnotes

*

Harris RP, Helfand M, Woolf SH, et al. Current methods of the US Preventive Services Task Force: a review of the process. Am J Prev Med 2001;20:21–35.