Abstract  The trustworthiness of individual studies is routinely characterized in systemic reviews by evaluating risk of bias, often by mechanistically applying standardized algorithms. However, such instruments prioritize the repeatability of the process over a more thoughtful and informative but necessarily somewhat more subjective approach. In mechanistic risk of bias assessments, the focus is on determining whether specific biases are present, but these assessments do not provide insights into the direction, magnitude, and relative importance of individual biases. In such assessments, all potential biases are naively treated as equally important threats to validity and equally important across all research topics, potentially leading to inappropriate conclusions about the overall strength of the available evidence. Instead, risk of bias assessments be should focused on identifying a few of the most likely influential sources of bias, based on methodologic and subject matter expertise, classifying each specific study on the basis of on how effectively it has addressed each potential bias, and determining whether results differ across studies in relation to susceptibility to each hypothesized source of bias. This approach provides insight into the potential impact of each specific bias, identifies a subset of studies likely to best approximate the causal effect, and suggests design features needed to improve future research.

Abstract  In recent decades, the fields of statistical and machine learning have seen a revolution in the development of data-adaptive regression methods that have optimal performance under flexible, sometimes minimal, assumptions on the true regression functions. These developments have impacted all areas of applied and theoretical statistics and have allowed data analysts to avoid the biases incurred under the pervasive practice of parametric model misspecification. In this commentary, I discuss issues around the use of data-adaptive regression in estimation of causal inference parameters. To ground ideas, I focus on two estimation approaches with roots in semi-parametric estimation theory: targeted minimum loss-based estimation (TMLE; van der Laan and Rubin, 2006) and double/debiased machine learning (DML; Chernozhukov and others, 2018). This commentary is not comprehensive, the literature on these topics is rich, and there are many subtleties and developments which I do not address. These two frameworks represent only a small fraction of an increasingly large number of methods for causal inference using machine learning. To my knowledge, they are the only methods grounded in statistical semi-parametric theory that also allow unrestricted use of data-adaptive regression techniques.

Abstract  When estimating causal effects using observational data, it is desirable to replicate a randomized experiment as closely as possible by obtaining treated and control groups with similar covariate distributions. This goal can often be achieved by choosing well-matched samples of the original treated and control groups, thereby reducing bias due to the covariates. Since the 1970s, work on matching methods has examined how to best choose treated and control subjects for comparison. Matching methods are gaining popularity in fields such as economics, epidemiology, medicine and political science. However, until now the literature and related advice has been scattered across disciplines. Researchers who are interested in using matching methods—or developing methods related to matching—do not have a single place to turn to learn about past and current research. This paper provides a structure for thinking about matching methods and guidance on their use, coalescing the existing research (both old and new) and providing a summary of where the literature on matching methods is now and where it should be headed.

Abstract  The goal of this article is to construct doubly robust (DR) estimators in ignorable missing data and causal inference models. In a missing data model, an estimator is DR if it remains consistent when either (but not necessarily both) a model for the missingness mechanism or a model for the distribution of the complete data is correctly specified. Because with observational data one can never be sure that either a missingness model or a complete data model is correct, perhaps the best that can be hoped for is to find a DR estimator. DR estimators, in contrast to standard likelihood-based or (nonaugmented) inverse probability-weighted estimators, give the analyst two chances, instead of only one, to make a valid inference. In a causal inference model, an estimator is DR if it remains consistent when either a model for the treatment assignment mechanism or a model for the distribution of the counterfactual data is correctly specified. Because with observational data one can never be sure that a model for the treatment assignment mechanism or a model for the counterfactual data is correct, inference based on DR estimators should improve upon previous approaches. Indeed, we present the results of simulation studies which demonstrate that the finite sample performance of DR estimators is as impressive as theory would predict. The proposed method is applied to a cardiovascular clinical trial.

Abstract  In observational studies for causal effects, treatments are assigned to experimental units without the benefits of randomization. As a result, there is the potential for bias in the estimation of the treatment effect. Two methods for estimating the causal effect consistently are Inverse Probability of Treatment Weighting (IPTW) and the Propensity Score (PS). We demonstrate that in many simple cases, the PS method routinely produces estimators with lower Mean-Square Error (MSE). In the longitudinal setting, estimation of the causal effect of a time-dependent exposure in the presence of time-dependent covariates that are themselves affected by previous treatment also requires adjustment approaches. We describe an alternative approach to the classical binary treatment propensity score termed the Generalized Propensity Score (GPS). Previously, the GPS has mainly been applied in a single interval setting; we use an extension of the GPS approach to the longitudinal setting. We compare the strengths and weaknesses of IPTW and GPS for causal inference in three simulation studies and two real data sets. Again, in simulation, the GPS appears to produce estimators with lower MSE.

Abstract  This paper considers conducting inference about the effect of a treatment (or exposure) on an outcome of interest. In the ideal setting where treatment is assigned randomly, under certain assumptions the treatment effect is identifiable from the observable data and inference is straightforward. However, in other settings such as observational studies or randomized trials with noncompliance, the treatment effect is no longer identifiable without relying on untestable assumptions. Nonetheless, the observable data often do provide some information about the effect of treatment, that is, the parameter of interest is partially identifiable. Two approaches are often employed in this setting: (i) bounds are derived for the treatment effect under minimal assumptions, or (ii) additional untestable assumptions are invoked that render the treatment effect identifiable and then sensitivity analysis is conducted to assess how inference about the treatment effect changes as the untestable assumptions are varied. Approaches (i) and (ii) are considered in various settings, including assessing principal strata effects, direct and indirect effects and effects of time-varying exposures. Methods for drawing formal inference about partially identified parameters are also discussed.

Abstract  Most causal inference methods consider counterfactual variables under interventions that set the exposure to a fixed value. With continuous or multi-valued treatments or exposures, such counterfactuals may be of little practical interest because no feasible intervention can be implemented that would bring them about. Longitudinal modified treatment policies (LMTPs) are a recently developed nonparametric alternative that yield effects of immediate practical relevance with an interpretation in terms of meaningful interventions such as reducing or increasing the exposure by a given amount. LMTPs also have the advantage that they can be designed to satisfy the positivity assumption required for causal inference. We present a novel sequential regression formula that identifies the LMTP causal effect, study properties of the LMTP statistical estimand such as the efficient influence function and the efficiency bound, and propose four different estimators. Two of our estimators are efficient, and one is sequentially doubly robust in the sense that it is consistent if, for each time point, either an outcome regression or a treatment mechanism is consistently estimated. We perform numerical studies of the estimators, and present the results of our motivating study on hypoxemia and mortality in intubated Intensive Care Unit (ICU) patients. Software implementing our methods is provided in the form of the open source R package lmtp freely available on GitHub (https://github.com/nt-williams/lmtp) and CRAN.

Abstract  Quantitative bias analysis serves several objectives in epidemiological research. First, it provides a quantitative estimate of the direction, magnitude and uncertainty arising from systematic errors. Second, the acts of identifying sources of systematic error, writing down models to quantify them, assigning values to the bias parameters and interpreting the results combat the human tendency towards overconfidence in research results, syntheses and critiques and the inferences that rest upon them. Finally, by suggesting aspects that dominate uncertainty in a particular research result or topic area, bias analysis can guide efficient allocation of sparse research resources. The fundamental methods of bias analyses have been known for decades, and there have been calls for more widespread use for nearly as long. There was a time when some believed that bias analyses were rarely undertaken because the methods were not widely known and because automated computing tools were not readily available to implement the methods. These shortcomings have been largely resolved. We must, therefore, contemplate other barriers to implementation. One possibility is that practitioners avoid the analyses because they lack confidence in the practice of bias analysis. The purpose of this paper is therefore to describe what we view as good practices for applying quantitative bias analysis to epidemiological data, directed towards those familiar with the methods. We focus on answering questions often posed to those of us who advocate incorporation of bias analysis methods into teaching and research. These include the following. When is bias analysis practical and productive? How does one select the biases that ought to be addressed? How does one select a method to model biases? How does one assign values to the parameters of a bias model? How does one present and interpret a bias analysis?. We hope that our guide to good practices for conducting and presenting bias analyses will encourage more widespread use of bias analysis to estimate the potential magnitude and direction of biases, as well as the uncertainty in estimates potentially influenced by the biases.

Abstract  Background: Exposure mixtures frequently occur in data across many domains, particularly in the fields of environmental and nutritional epidemiology. Various strategies have arisen to answer questions about exposure mixtures, including methods such as weighted quantile sum (WQS) regression that estimate a joint effect of the mixture components. Objectives: We demonstrate a new approach to estimating the joint effects of a mixture: quantile g-computation. This approach combines the inferential simplicity of WQS regression with the flexibility of g-computation, a method of causal effect estimation. We use simulations to examine whether quantile g-computation and WQS regression can accurately and precisely estimate the effects of mixtures in a variety of common scenarios. Methods: We examine the bias, confidence interval (CI) coverage, and bias–variance tradeoff of quantile g-computation and WQS regression and how these quantities are impacted by the presence of noncausal exposures, exposure correlation, unmeasured confounding, and nonlinearity of exposure effects. Results: Quantile g-computation, unlike WQS regression, allows inference on mixture effects that is unbiased with appropriate CI coverage at sample sizes typically encountered in epidemiologic studies and when the assumptions of WQS regression are not met. Further, WQS regression can magnify bias from unmeasured confounding that might occur if important components of the mixture are omitted from the analysis. Discussion: Unlike inferential approaches that examine the effects of individual exposures while holding other exposures constant, methods like quantile g-computation that can estimate the effect of a mixture are essential for understanding the effects of potential public health actions that act on exposure sources. Our approach may serve to help bridge gaps between epidemiologic analysis and interventions such as regulations on industrial emissions or mining processes, dietary changes, or consumer behavioral changes that act on multiple exposures simultaneously.

Abstract  In observational studies with exposures or treatments that vary over time, standard approaches for adjustment of confounding are biased when there exist time-dependent confounders that are also affected by previous treatment. This paper introduces marginal structural models, a new class of causal models that allow for improved adjustment of confounding in those situations. The parameters of a marginal structural model can be consistently estimated using a new class of estimators, the inverse-probability-of-treatment weighted estimators.

Abstract  In many randomized controlled trials the outcome of interest is a time to event, and one measures on each subject baseline covariates and time-dependent covariates until the subject either drops-out, the time to event is observed, or the end of study is reached. The goal of such a study is to assess the causal effect of the treatment on the survival curve. We present a targeted maximum likelihood estimator of the causal effect of treatment on survival fully utilizing all the available covariate information, resulting in a double robust locally efficient substitution estimator that will be consistent and asymptotically linear if either the censoring mechanism is consistently estimated, or if the maximum likelihood based estimator is already consistent. In particular, under the independent censoring assumption assumed by current methods, this TMLE is always consistent and asymptotically linear so that it provides valid confidence intervals and tests. Furthermore, we show that when both the censoring mechanism and the initial maximum likelihood based estimator are mis-specified, and thus inconsistent, the TMLE exhibits stability when inverse probability weighted estimators and double robust estimating equation based methods break down The TMLE is used to analyze the Tshepo study, a study designed to evaluate the efficacy, tolerability, and development of drug resistance of six different first-line antiretroviral therapies. Most importantly this paper presents a general algorithm that may be used to create targeted maximum likelihood estimators of a large class of parameters of interest for general longitudinal data structures.

Abstract  Ideally, questions about comparative effectiveness or safety would be answered using an appropriately designed and conducted randomized experiment. When we cannot conduct a randomized experiment, we analyze observational data. Causal inference from large observational databases (big data) can be viewed as an attempt to emulate a randomized experiment—the target experiment or target trial—that would answer the question of interest. When the goal is to guide decisions among several strategies, causal analyses of observational data need to be evaluated with respect to how well they emulate a particular target trial. We outline a framework for comparative effectiveness research using big data that makes the target trial explicit. This framework channels counterfactual theory for comparing the effects of sustained treatment strategies, organizes analytic approaches, provides a structured process for the criticism of observational studies, and helps avoid common methodologic pitfalls.

Abstract  The difference in difference (DID) design is a quasi-experimental research design that researchers often use to study causal relationships in public health settings where randomized controlled trials (RCTs) are infeasible or unethical. However, causal inference poses many challenges in DID designs. In this article, we review key features of DID designs with an emphasis on public health policy research. Contemporary researchers should take an active approach to the design of DID studies, seeking to construct comparison groups, sensitivity analyses, and robustness checks that help validate the method's assumptions. We explain the key assumptions of the design and discuss analytic tactics, supplementary analysis, and approaches to statistical inference that are often important in applied research. The DID design is not a perfect substitute for randomized experiments, but it often represents a feasible way to learn about casual relationships. We conclude by noting that combining elements from multiple quasi-experimental techniques may be important in the next wave of innovations to the DID approach.

Abstract  Clean air is fundamental to health. Compared to 15 years ago, when the previous edition of these guidelines was published, there is now a much stronger body of evidence to show how air pollution affects different aspects of health at even lower concentrations than previously understood. But here’s what hasn’t changed: every year, exposure to air pollution is still estimated to cause millions of deaths and the loss of healthy years of life. The burden of disease attributable to air pollution is now estimated to be on a par with other major global health risks such as unhealthy diets and tobacco smoking. In 2015, the World Health Assembly adopted a landmark resolution on air quality and health, recognizing air pollution as a risk factor for noncommunicable diseases such as ischaemic heart disease, stroke, chronic obstructive pulmonary disease, asthma and cancer, and the economic toll they take. The global nature of the challenge calls for an enhanced global response.

Abstract  This guidance has been developed as a basis for transparently characterizing uncertainty in chemical exposure assessment to enable its full consideration in regulatory and policy decision-making processes. Uncertainties in exposure assessment are grouped under three categories—namely, parameter, model and scenario—with the guidance addressing both qualitative and quantitative descriptions. Guidance offered here is consistent with other projects addressing exposure in the WHO/IPCS Harmonization Project, including a monograph on IPCS Risk Assessment Terminology, which includes a glossary of key exposure assessment terminology, and a monograph on Principles of Characterizing and Applying Human Exposure Models. The framework described in this monograph is considered applicable across a full range of chemical categories, such as industrial chemicals, pesticides, food additives and others. It is intended primarily for use by exposure assessors who are not intimately familiar with uncertainty analysis. The monograph aims to provide an insight into the complexities associated with characterizing uncertainties in exposure assessment and suggested strategies for incorporating them during human health risk assessments for environmental contaminants. This is presented in the context of comparability with uncertainties associated with hazard quantification in risk assessment. This document recommends a tiered approach to the evaluation of uncertainties in exposure assessment using both qualitative and quantitative (both deterministic and probabilistic) methods, with the complexity of the analysis increasing as progress is made through the tiers. The report defines and identifies different sources of uncertainty in exposure assessment, outlines considerations for selecting the appropriate approach to uncertainty analysis as dictated by the specific objective and identifies the information needs of decision-makers and stakeholders. The document also provides guidance on ways to consider or characterize exposure uncertainties during risk assessment and risk management decision-making and on communicating the results. Illustrative examples based on environmental exposure and risk analysis case-studies are provided. The monograph also recommends the adoption of 10 guiding principles for uncertainty analysis. These guiding principles are considered to be the general desirable goals or properties of good exposure assessment. They are mentioned in the text where most appropriate and are supported by more detailed recommendations for good practice. The 10 guiding principles are as follows: 1) Uncertainty analysis should be an integral part of exposure assessment. 2) The level of detail of the uncertainty analysis should be based on a tiered approach and consistent with the overall scope and purpose of the exposure and risk assessment. 3) Sources of uncertainty and variability should be systematically identified and evaluated in the exposure assessment. 4) The presence or absence of moderate to strong dependencies between model inputs is to be discussed and appropriately accounted for in the analysis. 5) Data, expert judgement or both should be used to inform the specification of uncertainties for scenarios, models and model parameters. 6) Sensitivity analysis should be an integral component of the uncertainty analysis in order to identify key sources of variability, uncertainty or both and to aid in iterative refinement of the exposure model. 7) Uncertainty analyses for exposure assessment should be documented fully and systematically in a transparent manner, including both qualitative and quantitative aspects pertaining to data, methods, scenarios, inputs, models, outputs, sensitivity analysis and interpretation of results. 8) The uncertainty analysis should be subject to an evaluation process that may include peer review, model comparison, quality assurance or comparison with relevant data or independent observations. 9) Where appropriate to an assessment objective, exposure assessments should be iteratively refined over time to incorporate new data, information and methods to better characterize uncertainty and variability. 10) Communication of the results of exposure assessment uncertainties to the different stakeholders should reflect the different needs of the audiences in a transparent and understandable manner.

Abstract  The Integrated Science Assessment (ISA) for Particulate Matter is a concise synthesis and evaluation of the most policy-relevant science, and has been prepared as part of the review of the primary (health-based) and secondary (welfare-based) National Ambient Air Quality Standards (NAAQS) for Particulate Matter (PM) under the Clean Air Act. Welfare effects are non-health effects which, according to the Clean Air Act include, but are not limited to, effects on soils, water, crops, vegetation, animals, wildlife and climate. The PM ISA, in conjunction with additional technical and policy assessments, provides the policy relevant scientific information necessary to conduct a review of the current primary and secondary air quality standards for particulate matter sufficiently protects public health and welfare. Particulate matter is one of six principal (or criteria) pollutants for which EPA has established National Ambient Air Quality Standards (NAAQS). The Clean Air Act requires EPA to periodically review the science for six major air pollutants, including PM. The Integrated Science Assessments (ISAs) summarize the science related to the health and welfare effects associated with these pollutants by providing a comprehensive review of the policy-relevant scientific literature published since the last NAAQS review. The ISA is a critical part of the scientific basis for updating the NAAQS. Impact/Purpose The PM ISA ERD reflects a systematic evaluation of the peer-reviewed literature published since the completion of the 2009 PM ISA that builds off the scientific conclusions presented in previous assessments of the health and welfare effects evidence for PM. This ISA is the initial draft of the current state of the science on the health and welfare effects of PM that will ultimately result in the final PM ISA that will serve as the scientific foundation of the ongoing review of the PM NAAQS, which was last completed in 2012.