An official website of the United States government
NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Yi Zhang, PhD, MS, Mae Thamer, PhD, Onkar Kshirsagar, MS, and Miguel Hernan, MD, DrPH.
Author Information and AffiliationsWhen making complicated health-related decisions, both patients and clinicians want to understand which treatments work best. The decisions may be supported by findings from randomized trials or, when they are not available, by the findings from observational data analyses that explicitly emulate a hypothetical randomized trial: the target trial.
We developed Comparative Effectiveness Research Based on Observational Data to Emulate a Target (CERBOT) Trial, a web-based tool that provides a structured standardized algorithm to define and emulate a target trial using observational data.
The research team designed and developed CERBOT using the following process: (1) stakeholders and expert engagement; (2) background literature review; (3) content development; (4) development of virtual wireframe and CERBOT website; and (5) alpha testing. We established the guiding principles for CERBOT development through a consensus process involving an 8-member advisory committee. As its basis, CERBOT uses a conceptual framework that identifies observational analysis as a means to emulate a target trial. This could be achieved by explicitly formulating the protocol of the target trial and addressing the feasibility of the conditions that must be met in order to emulate the target trial using observational data. We developed the actual CERBOT website through close collaboration with a web development firm using the most up-to-date technologies.
We developed an interactive user-friendly online tool, CERBOT.org. It includes 5 modules used to design and operationalize a comparative effectiveness research study by emulating a target trial. By synthesizing the information entered by users, CERBOT provides specific recommendations for causal inference analytical methods based on each user's individual research question. CERBOT also facilitates the formation and use of a multidisciplinary stakeholder research team. Through use of this research circle, it creates a communication platform for team members to collaboratively complete CERBOT modules by sharing ideas, comments, and conclusions.
CERBOT makes it easier for patients to participate in clinical research, by expressing their priorities and preferences in the design of the target trial, and assures researchers that their observational analysis is consistent with fundamental principles of causal inference.
More in-depth case studies are needed to demonstrate CERBOT's applicability for more complex interventions. Extensive testing in real-world settings is also warranted to assess merits, deficits, and feasibility of use.
To understand whether or how a therapy or intervention may affect a patient outcome, causal inference is often sought.1 For example, “What is the benefit or harm of using a new medication to ameliorate the outcomes of Alzheimer's disease?” is a causal question. While randomized clinical trials (RCTs), which determine the effects of various interventions on patient outcomes, remain the gold standard for answering such a question, they often are not ethically, practically, or economically feasible. In these cases, causal inference must be based instead on comparative effectiveness research (CER) studies that use observational data to provide timely answers.2,3 With the growth of electronic health records and increasing efforts to systematically collect information from routine clinical encounters, the potential for use of real-world observational data is exploding.4,5
Causal inference from observational data requires 2 steps6:
We developed a web-based tool called Comparative Effectiveness Research Based on Observational Data to Emulate a Target (CERBOT) Trial to help researchers specify and emulate target trials using observational data, a typically iterative process (Figure 1). Specifically, CERBOT does the following: (1) guides users to explicitly specify and emulate 5 critical components of the target trial protocol, which include eligibility criteria, treatment strategies, outcome of interest, follow-up period, and causal contrast(s) of interest; (2) helps users select appropriate methods for analysis; and (3) provides a structure for stakeholders and methodologists to engage in both specification and emulation of the target trial.
CERBOT Flowchart Depicting Completion of 5 Modules.
This report describes the features of CERBOT (http://CERBOT.org), which is designed to be used by researchers and clinicians with basic statistical training, who will work with selected teams of other researchers and stakeholders. Depending on the specific question, such stakeholders could include clinicians, policy makers, regulators, patients, patient advocates, caregivers, third-party payers, and manufacturers.
The research team designed and developed CERBOT over 3 years using an approach comprising 5 major parts: (1) stakeholders and expert engagement; (2) background research; content development; (4) development of virtual wireframe and CERBOT website; and (5) conduct of in-house and CITAC testing (alpha testing).
The research team developed CERBOT through a consensus process that involved a group of academic, industry, and clinician researchers. This consultation group was tasked with reviewing feasible frameworks to help in the development and implementation of a causal inference toolkit. At the beginning of the project, the research team formed an 8-member Causal Inference Toolkit Advisory Committee (CITAC) that included clinicians, applied researchers, elite statisticians, experts in dissemination and implementation fields, and leaders in the pharmaceutical industry (Table 1).
CITAC Members.
We achieved consensus through the nominal group technique (NGT), which is a structured variation of a small-group discussion to reach consensus.12,13 The NGT process prevents the domination of discussions by a single person or opinion leader and instead encourages all group members to participate; it results in a set of prioritized solutions or recommendations that represent the group's preferences. The NGT also has the advantage of diminishing competition and pressure to conform based on the status within a group. Specifically, we utilized the general 4-step process to conduct the NGT as follows:
The remainder of this section provides details on how the NGT was implemented. The research team's first CITAC in-person daylong meeting took place on March 21, 2014. There the research team presented to CITAC the objectives of the project and introduced general conceptual steps for conducting causal inference using observational data. Globally, the research team outlined and discussed 3 missions of the causal inference toolkit: (1) Educate stakeholders in the development of a well-defined research question via interactive step-by-step instructions on how to emulate a randomized hypothetical trial using observational data; (2) provide guidance on the selection and use of CER techniques—especially causal modeling techniques; and (3) foster the collaboration of various expert stakeholders with a wide variety of complementary skill sets.
CITAC members provided important feedback regarding the aim, scope, functionality, and target audience of the toolkit; overall framework for building the toolkit; and ways to make the toolkit more user friendly and accessible. The team and CITAC also discussed and agreed on the following 4 guiding principles for toolkit development:
During the second day of the meeting, the research team and CITAC agreed on using the target trial framework as the conceptual basis for developing CERBOT. Attendees further discussed both traditional and nontraditional ways to make the tool more user friendly and patient centered. For example, committee members emphasized that the toolkit needed to be welcoming, not intimidating, at every stage of the process; examples should always be provided to illustrate complex concepts or ideas; didactic content should be succinct and to the point; and the toolkit should be easy to access and navigate and should have interactive features.
The research team and CITAC agreed that the optimal format of the toolkit should be an interactive, cloud-based website that would house the learning modules, examples, and reporting functionality. Given the scientific nature of the website, CITAC suggested that the target audience for the website would be health care and/or clinician researchers who had previous experience with epidemiological investigations. The tool will be particularly useful to researchers who are currently using traditional noncausal techniques and until recently have lacked guidance in how to conduct causal inference studies, but who, now, may consider shifting to causal inference studies. By following the structure and template provided by CERBOT, users are assured that their CER study design is consistent with the fundamental principles of CER for causal inference.
Based on the committee's feedback, particularly regarding identification of CERBOT's target audience as researchers with basic statistical knowledge, the team developed the preliminary website wireframe and content pages, and distributed related documents to CITAC on January 15, 2015. Based on CITAC's subsequent feedback, the research team finalized the main features of CERBOT and shared them with CITAC members in June 2015.
Following Years 2 and 3, the focus was delineating CERBOT requirements regarding user experience and building the CERBOT website based on the defined requirements. During these 2 years, the focus was 2-fold: technological development of the website itself, by collaboration with a web development subcontractor; and continued communication with committee members through regular contact to discuss incremental progress of the website development as well as its specific content. The advisory committee led the effort to promote user-friendliness, as general clinicians may have limited knowledge regarding causal inference but are interested in learning more about using it to conduct their CER research.
During September and November of 2016, a preliminary version of the tool was piloted within CITAC. The research team asked the committee to evaluate the user-friendliness of the site and to assess the content of the website itself, specifically its value and utility as a resource for a CER team. Committee members provided their views and suggestions on the potential utility of the modules, interactive features, and navigation of CERBOT. After addressing CITAC's concerns, the team made CERBOT version 1.0 available to the public in January 2017. By actively engaging CITAC members in the process of developing the causal inference toolkit, the project ensured credibility, transparency, user-friendliness, and enhanced value for a wide variety of stakeholders.
The research team conducted background research to help establish guiding principles and to refine the conceptual framework for the development of CERBOT. As described in the Background section, the basic idea behind CERBOT is that CER observational studies need to emulate hypothetical RCTs as closely as possible. Thus, the background research included a literature review of articles published in academic journals. Specifically, we performed a PubMed search for English-language articles from 1975 to January 2017, using the following keywords: causal inference framework, target (or hypothetical) trial, and emulation. We grouped articles into 2 categories.
The first category included review studies that introduced and discussed the need to frame research questions as hypothetical randomized trials in order to make them directly relevant for decision making. Hernán and Robins outlined a target trial framework and emphasized that causal analyses of observational data need to be evaluated in terms of how well they emulate a particular target trial.8
The second category of articles included actual CER studies that explicitly or inexplicitly followed the target trial framework in their design and analysis of observational studies for comparative effectiveness and safety. Prior explicit attempts to emulate trials using observational data have studied, for example, postmenopausal hormone therapy,14 statins,15 epoetin,16,17 screening colonoscopy,18 and antiretroviral therapy.19 Some studies followed the target trial framework to analyze observational data, although the target trial was not explicitly described.18,20 Review of the literature helped the team envision how specific steps within this general framework should be constructed for CERBOT users to follow.
Additionally, the research team conducted a search on best CER practices, reporting guidelines, methodology standards, and frameworks related, directly or indirectly, to the design and implementation of causal inference methods. The purpose of the scan was to better understand existing work, to ensure that the development of CERBOT was evidence based, and to help identify the original and unique features of CERBOT.21-33
The content builds on the framework that the project team and CITAC developed for the CER study design and CER analysis. Consistent with the framework, CERBOT guides users to conduct their observational analyses according to the following steps: (1) engage relevant stakeholders to work collaboratively on the CER question; (2) articulate the research question in terms of a hypothetical randomized trial (the target trial); (3) outline how to emulate the target trial using observational data; and (4) select suitable analytical methods based on user's specific research question.
CERBOT supports the full scope of CER questions. Our premise is that any CER question can be articulated in terms of a hypothetical target trial. If a target trial cannot be proposed, then the comparative effectiveness question is deemed ill defined. The use of the target trial framework ensures that the research focuses on the effects of interventions that occur in the data and are well defined.21-23
The intention was for the tool to be useful to researchers and others with varying levels of experience in causal inference research, and for the text to be informative and instructive for both neophytes and experts. For example, when guiding users to emulate the target trial, a template is provided to facilitate the process. Project leaders prepared the first draft of CERBOT pages, using the results of their background research as well as communication with CITAC. Then all project team members reviewed and edited CERBOT pages for content and structure. The investigative team conducted past and ongoing content management, continuously making small-scale modifications.
Since July 2016, the project team has been working closely with a web development team to create the actual website that will deliver the required functionalities. The team used Agile methodology to work with the web development team in the virtual wireframe and website development. During this process, the investigative team oversaw the entire effort, including design, development, deployment, regression testing, bug fixing, and performance testing, by maintaining a close working relationship with the web development team. Specifically, the project team conducted in-person meetings and conference calls with the web development team to share user stories and to help team members understand project requirements. The project team also communicated closely with web development team to ensure appropriate design development, project progress, and on-time delivery.
For website quality control, the project team tested CERBOT's features and used the Jira tracking system21 for compiling and monitoring reported problems. Subsequently, the research team and developers documented, discussed, and rectified most of the issues recorded in Jira. Finally, we set up a dedicated server responsible for incorporating the latest changes and updates, known as a build server, for the developers to ensure constant availability of the latest revised version.
To assess the functionality of CERBOT and to ensure that it met all project objectives, the research team began testing CERBOT in December 2016 after it was first released by the developers to the research team. We conducted 2 stages of alpha testing. Members from the project team, including the principal investigator (PI), co-PI, and 2 other key personnel, conducted comprehensive testing of CERBOT functionalities during the first stage. Defects and bugs were identified and remedies were prioritized during several iterations of testing. At the completion of the first testing stage, we held a meeting with the website developer, during which we used a structured questionnaire to determine unmet requirements regarding CERBOT flow and functionality. Considering budget limits, the research team prioritized issues to be addressed and then worked closely with the web development team to fix the 5 most critical defects: navigation, parallel process of specification and emulation, research circle functionality, interactive features, and reporting functionalities.
After the main defects were fixed, the research team started the second stage of alpha testing by releasing CERBOT to CITAC and instructing CITAC members to conduct 1 to 2 hours of testing. Because CITAC members were involved in conceptualizing CERBOT (although they were not involved directly in actual development of the CERBOT website), we define testing conducted by expert committee members as alpha testing. Three committee members provided feedback, which focused primarily on 3 areas:
Building on the conceptual framework of the target trial approach, the following features formed the core elements of CERBOT.
To foster collaboration between researchers and stakeholders, CERBOT facilitates joint work by a research team to complete the modules and includes, for each module, the format for a discussion section among research team members. This allows for the creation of a research circle, made up of all team members working on a single research question, and enables members to interact and share ideas about CER study design and analysis.
Each collaborative research circle is led by the user who created the research question using CERBOT. To build a research circle, a user invites other people to join the team and collaborate on a specified research question via email exchanges. Invitees join the circle via a link provided in the email text. Research circle members can then jointly and concurrently work on completing the modules. For example, if a researcher does not know how to define acute kidney failure, using ICD-9 diagnosis or procedure codes, the researcher has 3 options: (1) Notify a team clinician of this in the “Concerns” box; (2) raise the issue in the “Discussion” section following the module report; or ask the question directly via email message in the “Research Circle” box included in CERBOT. Modules and reports are then automatically updated after a team member addresses and resolves the concern. In addition, team members can post/create topics and comments in the discussion section linked to specific modules.
In Figure 1, the CERBOT flowchart illustrates the process of specifying and emulating the target trial. Five essential CERBOT modules—eligibility criteria, study outcomes, follow-up, treatment strategies, and causal contrast and adjustment variables—correspond to 5 key components of the protocol of the target trial. Each module requires 3 types of input from the users:
For example, if the eligibility criteria require patients to have both normal hemoglobin and cholesterol levels at the baseline, but hemoglobin information is not included in the data set, the user repeats the process with new input; for example, redefining the target population, by dropping the requirement for hemoglobin levels, may achieve better results.
CERBOT is structured to support users by facilitating their specification and emulation of the target trial through a side-by-side and iterative process. If users need to learn about the meaning of a certain component or question, they can read the helper text under each component or click on the “Tip” link provided. A small pop-up window opens with instructions, examples regarding completion of the specific component, and answers to specific questions. Users can view, revise/edit, save module updates, and continue to research previously saved questions. Two case studies included in CERBOT show examples using the complete process, from the initial specification, emulation, and resolving of concerns through the report, any revisions, and results.
Using the information entered by the user in previous modules, CERBOT recommends appropriate causal inference approaches for data analysis. Specifically, CERBOT asks 7 questions to recommend a specific method (Figure 2). It is important to note that CERBOT is not intended to serve as a data-processing or computing software that constructs analytical data or calculates estimated risks. For implementation of CERBOT-recommended methods, a brief tutorial and pertinent literature from the “CERBOT Resource” page are provided on the website. The 7 questions driving the decision regarding appropriate analytical methods are the following:
Algorithms used by CERBOT for recommending analytical methods based on answers to 7 questions from CERBOT users.
CERBOT “reacts” or responds to the individual user's specific input. Its dynamic structure picks up the user's choices and accordingly structures the study design, adjustment variables, and choice of causal inference methods. For example, if the answer to the question “Do you expect that censoring may result in selection bias?” in module 3 (Study Follow-up) is “Yes,” then CERBOT prompts users to define a time-varying variable in module 5. This added variable is then included in the analysis plan to account for loss to follow-up and/or other censoring events.
Each module produces an individual report, which is a summary of all the questions/answers in the current module. Each automatic module report is intended to serve as the basis for discussion among research team members and illuminates any subsequent revisions of the input. Users can view or download this report after submitting answers to required questions. Upon completion of modules 1 to 5, CERBOT users are provided with a final report page summarizing the entire study design. Also provided are recommended methods that include standard statistical methods as well as new methods developed for causal inference that use observational data, such as g-methods,33 instrumental variables,47 and doubly robust methods35,48 (Figure 2). These reports are downloadable, can be edited in Microsoft Word format, and can be shared among the entire research team.
The research team created a short tutorial video, linked to the CERBOT website, to help users understand the overall functionality of CERBOT. The tutorial provides an overview of the main objectives of CERBOT, how to create a research question, how to invite team members as part of the research circle, and how generally to navigate the website. To illustrate how each module should be completed, the research team added 2 case studies based on the team's published research,17,49 using sustained interventions that explicitly follow the framework for specifying and emulating a target trial. Each case study showcases how the specification and emulation process should be completed and reported. Case studies can be browsed by clicking the “Case Studies” button in the sidebar of each module page.
We conducted many limited dissemination activities to identify potential CERBOT users and familiarize them with the concepts and functionality of the causal inference website. First, Yi Zhang, PhD, MS, project PI, gave 2 presentations on CERBOT use and functionality at the Joint Statistical Meeting in 2015 and 2016, respectively. Her presentations generated interest among conference attendees, who are currently awaiting the final finished website. Second, at the PCORI-funded Causal Inference Methods for PCOR Using Observational Data (CIMPOD) meeting held in February 2017, we set up a booth to allow conference attendees—most of whom were statisticians or clinical researchers with interest in using causal methods—to “test” CERBOT and provide feedback. Overall, we found tremendous interest in using CERBOT among CIMPOD attendees; their feedback included the need for case studies as well as a tutorial to provide guidance in the use of CERBOT for a novice user. The research team has subsequently addressed these suggestions. Third, we have introduced CERBOT to 5 sites comprising 11 research teams that we are currently working with under a PCORI dissemination and implementation (D&I) project to disseminate use of g-methods to researchers with ongoing CER questions. These teams are currently in the process of receiving tutorials in the use of g-methods, particularly the more complex g-formula, and hope to use CERBOT after this foundational work to augment their current software and research tools.
We built the CERBOT website on NodeJS, React, and Redux technologies with MongoDB as database configuration and Amazon EC2 as the production server. The maintenance of CERBOT includes updates and changes to the contents of the website, troubleshooting, and data backup and archiving using Amazon Web Services. The production website URL (http://www.cerbot.org) serves as the CERBOT tool. It can be accessed through standard web browsers running in various operating systems and hardware platforms. CERBOT is available at no cost to the public and can be distributed or shared freely at any time, provided the original work is properly cited and the use is noncommercial.
The key concept underlying CERBOT is that observational analyses can be viewed as an endeavor to emulate a hypothetical randomized trial—the target trial—to answer a causal question that is not feasible for RCTs.9 The target trial approach is consistent with the formal counterfactual theory of causal inference,50,51 provides an organizing principle for causal inference methods that implicitly rely on counterfactual reasoning,52,53 and prevents common methodological pitfalls, such as immortal time bias and selection bias. The target trial approach also facilitates a systematic and transparent evaluation of the observational study design9 and thus promotes better communication and understanding. The utility of this causal inference framework has been demonstrated in a variety of applications and across many disease areas.29,54-56
CERBOT implements the target trial approach primarily by guiding users to complete 5 modules included in CERBOT. Consistent with existing study design and reporting guidelines,57-61 as well as PCORI Methodology Standards. these 5 modules are structured to support the parallel process of specifying and emulating the target trial for the question at hand. Many investigators, for example, are unfamiliar with the idea of emulation of a target trial and may have trouble trying to implement it in practice. CERBOT helps investigators make the emulation explicit, by breaking up this complicated concept into easy-to-follow steps. For example, necessary components listed in each module are accompanied by instructions and tips to guide researchers or clinicians. This allows people with limited knowledge of causal inference to design a study and select an analytic approach that is consistent with causal inference principles. Researchers can continue to apply the structure provided by CERBOT to clarify the target trial and to organize information about available data to emulate the trial.
CERBOT can be used as a complement to several other causal inference introductory tools and/or educational materials that now exist, for use in observational data to better assess causality and to allow for less biased estimates of effect. Multiple tools that address many CER topics and are presented in a variety of ways include tutorials, conferences62 and workshops,63 presentation slides, videos, short courses, webinars, and software supports. For example, the complete online course “Introduction to Causal Inference” (www.ucbbiostat.com) by Petersen and Balzer provides a roadmap and necessary steps for causal inference studies using observational data; journal articles by Greenland,37 Ahern et al,64 and Naimi33 provide tutorials on the methods presented in CERBOT.
Compared with existing introductory tools for causal inference, CERBOT has a unique focus on explicitly defining causal questions and provides an approach to emulate a target trial within an observational setting. An innovative feature of CERBOT is that it is organized as a collaboration site for stakeholders and researchers to jointly contribute their knowledge and expertise.
The research team recognizes several challenges in promoting and disseminating CERBOT for designing CER and implementing causal inference methods. Most important, CERBOT is still in its inaugural phase and an extensive assessment for its feasibility in real-world settings is needed. Such an assessment would include a collection of case studies that illustrate, by detailed examples, how to explicitly emulate a target trial using CERBOT. Further adaptations may be useful to refine the components included in each module or flowchart used for selecting appropriate methods. To date, the CERBOT development process has involved only project investigators and CITAC committee members. Although all committee members are notable researchers in the fields of CER and public health, successful promotion and dissemination of CERBOT must include other researchers, methodologists, and statisticians across various health and related research fields. To enhance future updates of CERBOT, we will apply for a PCORI D&I project to conduct a pseudo-experimental study in order to assess and compare study designs and outcomes, among a cohort of researchers using CERBOT, vs contemporaries who use another educational platform. In future efforts, we propose the inclusion of CERBOT in introductory statistical courses as well as its use in biostatistical departments across various academic centers. Finally, we aim to present CERBOT capabilities and functionalities at upcoming meetings regarding statistical and health services research, and patient outcomes issues.
Several limitations are noteworthy. Most notably, extensive testing for feasibility in real-world settings is warranted to assess both merits and deficiencies of using CERBOT. While CERBOT presents complex topics in simple language for purposes of accessibility and understanding, users may be required to review material elsewhere for more in-depth understanding and project specificity. Finally, additional case studies that present complex dynamic interventions are required and are not currently available to permit users to better understand design and analysis of such studies using observational data.
Future work will focus on adding components to CERBOT to improve ease of use, integrate other existing CER tools into CERBOT, and enhance interactive features to improve the decision tree for choosing appropriate methods. Planned tool developments include the following:
Nonrandomized, or observational, studies are critical in proving cause and effect for treatments and patient outcomes. Such studies provide evidence about real-world populations and practices that complement the information available from RCTs. In some cases, observational studies present data not otherwise available. The research team developed a user-friendly web application to improve CER study design and analysis by means of explicitly emulating a target trial. The causal inference framework, developed by the project co-PI and his colleagues, is essentially the ability to conceptualize observational studies for comparative purposes and as attempts to emulate randomized experiments. Key features of CERBOT include steps that are easy to follow, in order to formulate a detailed study design; recommendations for analytical methods, based on specific types of research questions and specific types of biases and confounding issues (as also recommended by PCORI Methodology Standards); and facilitated teamwork among researchers and stakeholders, who include researchers, clinicians, patients, and others suitable for addressing specific PCOR CER questions.
In the current era of big data, the use of CERBOT—in combination with subject matter expertise, epidemiologic and methodologic proficiency, and innovative computer science tools—can help maximize the societal benefits of big data for causal inference.
Research reported in this report was [partially] funded through a Patient-Centered Outcomes Research Institute® (PCORI®) Award (#ME-1303-6031) Further information available at: https://www.pcori.org/research-results/2013/developing-interactive-online-guide-support-use-causal-inference-methods
Zhang Y, Thamer M, Kshirsagar O, Hernan M. (2019). Developing an Interactive Online Guide to Support the Use of Causal Inference Methods in Comparative Effectiveness Research. Patient-Centered Outcomes Research Institute (PCORI). https://doi.org/10.25302/1.2020.ME.13036031
The [views, statements, opinions] presented in this report are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute® (PCORI®), its Board of Governors or Methodology Committee.
This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License which permits noncommercial use and distribution provided the original author(s) and source are credited. (See https://creativecommons.org/licenses/by-nc-nd/4.0/
Your browsing activity is empty.
Activity recording is turned off.
See more...
