OHDSI Home | Forums | Wiki | Github

[2022 US Symposium] Phenotype Development and Evaluation Workgroup activity

General
#1

:rotating_light: Phenotype Development and Evaluation Workgroup activity :rotating_light:

Phenotyping enthusiasts, please join the Phenotype Development and Evaluation Workgroup for our activity at the 2022 OHDSI US Sympoisum! This workgroup activity will be facilitated by myself, @jweave17, @Gowtham_Rao, and @AzzaShoaibi/@Azza_Shoaibi. Please see the details below and share any comments you may have.

When:

Goal:

  • Extend the terrific work done by the community during Phenotype Phebruary
  • Formalize the process by which computable phenotypes are developed, evaluated, and importantly, validated

Background:

  • Established, status quo, gold-standard validation studies are of immense value but have limitations given current observational research context of large, disparate, distributed database networks of heterogenous populations around the world
  • Current validation approaches generally assess one phenotype in one data source/study population at a time. These provide high internal validity, but questionable external validity
  • Involves manual chart review by a clinical expert, which limits number of case adjudications possible in a validation study. We get high quality results, but are time and resource intensive
  • They generally assess cases only, which provides PPV but often with NPV or sensitivity, which is an incomplete picture of phenotype definition operating characteristics
  • The validation metrics are often applied to external, possibly non-applicable study populations, which is problematic given that we know measurement error correction is highly sensitive to measurement error accuracy and applicability: it’s not well understood how well measurement errors transport across populations

Opportunity:

  • The OHDSI community has the people, the data, and the tools to advance computable phenotype validation by formalizing the process, born in Phenotype Phebruary, to solve some of these problems
  • We’d like to invite data-holders to participate in a workshop where we will introduce a 2-part OHDSI validation study approach: the combination of 1) comprehensive clinical characterization [CohortDiagnostics] in conjunction with 2) probabilistic reference standard validation [PheValuator]
  • We’d like your participation in the development this novel method for rapid phenotype development, evaluation, and validation that is scalable across a data network. The intent is that internal phenotype validation result can feasibly be available for any study using data in the OHDSI network
  • Participants will run a study package that will characterize and validate an established phenotype in advance of the workshop and will report results during the workshop
  • We will interpret and compare results with existing validation studies of the established phenotype

What next?

  • Stay tuned for information on which traditionally validated phenotype we’ll evaluate and validate during the workshop
  • The pre-workshop study package and execution instructions will follow
2022 OHDSI Workgroup Activities: How Do I Choose?
#2

:rotating_light: Phenotype Development and Evaluation Workgroup activity UPDATE :rotating_light:

Dear Phenotype Dev and Eval Workgroup activity participants,

Are you a data holder with a CDM and would like to evaluate 5 phenotype algorithms intended to identify patients with acute pancreatitis in your database? Are you interested in contributing results to an exercise that will demonstrate the extent to which acute pancreatitis clinical characteristics and measurement errors vary by phenotype algorithm and by database?

If you’ve answered yes :star_struck: to these questions, then please navigate to ApPhenotypeEvaluation to run the analysis and share results in advance of the workshop. By doing so, you’ll be participating in one of the first efforts to implement a novel, multi-database, probabilistic reference standard approach to phenotype evaluation and validation that has the potential to estimate trustworthy phenotype measurement errors at scale.

To execute, please follow the study execution instructions exactly. If you’ve registered for the workshop, please expect an email from me, @jweave17, with some further instructions on how to troubleshoot if you experience any issues loading or running the analysis package. Huge thanks to @aostropolets and @Evan_Minty for their help testing this!

Lastly, if you’re so inclined, please see some related material worth reading below. This isn’t required, but it will provide some helpful context for the workshop.

cc: workshop facilitators @Gowtham_Rao, @AzzaShoaibi/@Azza_Shoaibi, @jswerdel

t