OHDSI Home | Forums | Wiki | Github

[2022 US Symposium] Phenotype Development and Evaluation Workgroup activity

:rotating_light: Phenotype Development and Evaluation Workgroup activity :rotating_light:

Phenotyping enthusiasts, please join the Phenotype Development and Evaluation Workgroup for our activity at the 2022 OHDSI US Sympoisum! This workgroup activity will be facilitated by myself, @jweave17, @Gowtham_Rao, and @AzzaShoaibi/@Azza_Shoaibi. Please see the details below and share any comments you may have.



  • Extend the terrific work done by the community during Phenotype Phebruary
  • Formalize the process by which computable phenotypes are developed, evaluated, and importantly, validated


  • Established, status quo, gold-standard validation studies are of immense value but have limitations given current observational research context of large, disparate, distributed database networks of heterogenous populations around the world
  • Current validation approaches generally assess one phenotype in one data source/study population at a time. These provide high internal validity, but questionable external validity
  • Involves manual chart review by a clinical expert, which limits number of case adjudications possible in a validation study. We get high quality results, but are time and resource intensive
  • They generally assess cases only, which provides PPV but often with NPV or sensitivity, which is an incomplete picture of phenotype definition operating characteristics
  • The validation metrics are often applied to external, possibly non-applicable study populations, which is problematic given that we know measurement error correction is highly sensitive to measurement error accuracy and applicability: it’s not well understood how well measurement errors transport across populations


  • The OHDSI community has the people, the data, and the tools to advance computable phenotype validation by formalizing the process, born in Phenotype Phebruary, to solve some of these problems
  • We’d like to invite data-holders to participate in a workshop where we will introduce a 2-part OHDSI validation study approach: the combination of 1) comprehensive clinical characterization [CohortDiagnostics] in conjunction with 2) probabilistic reference standard validation [PheValuator]
  • We’d like your participation in the development this novel method for rapid phenotype development, evaluation, and validation that is scalable across a data network. The intent is that internal phenotype validation result can feasibly be available for any study using data in the OHDSI network
  • Participants will run a study package that will characterize and validate an established phenotype in advance of the workshop and will report results during the workshop
  • We will interpret and compare results with existing validation studies of the established phenotype

What next?

  • Stay tuned for information on which traditionally validated phenotype we’ll evaluate and validate during the workshop
  • The pre-workshop study package and execution instructions will follow