Topic: Reliability in Observational Research: Assessing Covariate Imbalance in Small Studies
Presenter: George Hripcsak, Vivian Beaumont Allen Professor of Biomedical Informatics, Columbia University
Abstract: One of the major challenges facing observational research is the risk of producing a biased estimate due to confounding. Propensity score adjustment, invented 40 years ago, addresses confounding by balancing covariates in subject treatment groups through matching, stratification, inverse probability weighting, etc. Diagnostics ensure that the adjustment has been effective. A common technique is to check whether the standardized mean difference for each relevant covariate is less than a threshold like 0.1. For small sample sizes, the probability of falsely rejecting a study because of chance imbalance where no underlying balance exists approaches 1. We propose an alternative diagnostic that checks whether the standardized mean difference statistically significantly exceeds the threshold. With simulation and real-world data, we find that this diagnostic achieves a better trade-off of type 1 error rate and power than nominal threshold tests and than not testing for sample sizes of 250 to 4000 and for 20 to 100,000 covariates. In network studies, meta-analysis of effect estimates must be accompanied by meta-analysis of the diagnostics or else systematic confounding may overwhelm the estimated effect. Our procedure for statistically testing balance at both the database level and the meta-analysis level achieves the best balance of type-1 error rate and power. Our approach supports the review of large numbers of covariates, enabling a more rigorous diagnostic process.
Bio: Dr. George Hripcsak, MD, MS, is Vivian Beaumont Allen Professor at Columbia University’s Department of Biomedical Informatics. Dr. George Hripcsak, MD, MS, is Vivian Beaumont Allen Professor at Columbia University’s Department of Biomedical Informatics. He is a board-certified internist with degrees in chemistry, medicine, and biostatistics. Dr. Hripcsak is interested in the clinical information stored in electronic health records and their use in improving health care. Health record data are sparse, irregularly sampled, complex, and biased. Using causal inference, nonlinear time series analysis, machine learning, knowledge engineering, and natural language processing, he is developing the methods necessary to produce reliable medical evidence from the data. Dr. Hripcsak leads the Observational Health Data Sciences and Informatics (OHDSI) coordinating center; OHDSI is an international network with thousands of collaborators from 83 countries and health records on almost one billion unique patients. He co-chaired the Meaningful Use Workgroup of the U.S. Department of Health and Human Services’ Office of the National Coordinator of Health Information Technology. Dr. Hripcsak is a member of the National Academy of Medicine, the American College of Medical Informatics, the International Academy of Health Sciences Informatics, and the New York Academy of Medicine. He was awarded the highest honor by the American College of Medical Informatics, the 2022 Morris F. Collen Award of Excellence. He has over 500 publications.