OHDSI Home | Forums | Wiki | Github

[Population-Level Estimation] Method Evaluation Task Force

I’m hereby announcing the formation of the Method Evaluation Task Force!

When designing a study there are many different study designs to choose from, and many additional choices to make, and it is often unclear how these choices will affect the accuracy of the estimate.(e.g. If I match on propensity scores, will that lead to more or less bias than when I stratify? What about power?) The literature contains many papers evaluating one design choice at a time, but (to me) with unsatisfactory scientific rigor; often a method is evaluated on one or two exemplar study from which we cannot generalize, or by using simulations which have an unclear relationship with the real world. The OMOP Experiment was a first attempt at systematic empirical evaluation of method performance, from which we have learned many insights (mostly on how to better evaluate methods :wink: ).

Task force objectives

  1. Develop the methodology for evaluating methods (for estimating population-level effects).
  2. Use the developed methodology to systematically evaluate a large set of study designs and design choices.

Next steps

  1. Call for collaborators (this post)
  2. Some further refinement of the objectives (e.g. what gold standards to use, which methods to include in the evaluation)
  3. Gather requirements for funding, and apply for funding if needed
  4. Do the research

Call for collaborators
Let us know if you’re interested in actively participating in this research. Warning: this will be hard work with (probably) no pay. Either send me an e-mail, or respond to this post.

Tentative members of this task force are currently: @nicolepratt, @jweave17, Alejandro Schuler, @nigam, @yuxitian, and me.


Sure, sign me up!

I am interested in the Method Evaluation Task Force. George

I am interested in this group, Teng

Please, sign me up!

I would like to re-invest in OHDSI. I would like to join the task force and after understanding how you propose to go about this evaluation will make a time commitment. I do think a conceptual paper mapping epidemiologic design constructs to database epidemiology is needed. For example, we need to standardize terminology, provide rationale and example operationalization. One example I like to use is the baseline concept. In a clinical trial a full clinical evaluation is done at enrollment or at study index. Why do we use a baseline concept and how what purpose does it serve in relation to a clinical trial?

Maybe you have already done this but I have not found a paper mapping these concepts to database epidemiology. Some concepts for consideration:
Study period
Wash out
Incident treatment or incident course of therapy vs. inception cohort
Incident disease can also be understood from a chronic disease epi or infectious disease epi perspective
Index date

Validation and how information on measurement performance can be used to estimate “true prevalence” vs apparent prevalence. And how we encorporate measurement performance into our analytic framework. There are so many issues for database epidemiology that have not been pulled together into one paper or series that would provide as a tutorial with justification for how to implement a specific study design

Hi @brian_s!

The task force will focus on evaluation of population-level effect estimation, so not on prevalence estimation. The focus is on methods that try to estimate the causal effect of an exposure on an outcome (although I realize the term ‘causal’ is a loaded term).

The approach we’ll use is empirical evaluation, so using some gold standard set of exposure-outcome pairs where the true effect is known to measure how well methods perform. Even though I agree that the terms you mention need explicit definitions, I would prefer the task force focus on the evaluation aspect, and simply provide its own definitions (and source code) of the designs under evaluation.

Please sign me up. I am not sure how to contribute to this task force, however at least we are able to run the tools against to our data.

Welcome Rae! You should now have access to the restricted task force forum.

Martijn please put me on the list we can help out from Erasmus

Please sign me up, as well.

Mike Goodman
AstraZeneca Patient Safety

I’d also like to help with this task force. Thanks!

Hello, I recently joined OHDSI and look forward to contributing to this awesome initiative! I would like to be a research collaborator. Thanks!


1 Like