This paper aims to provide a structured comparison of OHDSI’s recommended phenotype development and evaluation practices with major global regulatory guidelines.
Why This Matters
Phenotyping is crucial to RWE credibility:
Regulatory alignment: Our early findings indicates concordance between OHDSI’s approach and global guidelines.
Open-science synergy: The community-driven nature of OHDSI can support transparent and reproducible phenotyping.
Key Points From Our Draft
Phenotype Development
Importance of conceptual (clinical) and operational (coded) definitions.
Phenotype Evaluation
Need to quantify misclassification (sensitivity, specificity, PPV, NPV, etc.).
Utility of OHDSI tools like CohortDiagnostics and PheValuator for empirical assessment.
Call for Collaborators
We are currently refining the manuscript and invite interested OHDSI community members to:
Provide additional examples or case studies where OHDSI phenotyping was used in a regulatory context.
Review and comment on sections related to specific regulatory guidance’s (FDA, EMA, PMDA, etc.).
If you are interested in collaborating, please leave a reply below or email us directly or discuss in OHDSI Phenotype Development and Evaluation Workgroup.
You may want to dissect the guidelines from various regulators. After all, they are all pretty fluffy. Only the FDA one is much more specific. Comparing and evaluating them is probably a good thing and helps the audience.
You need to declare our framework and give it a distinct brand. And stick to it. “OHDSI’s community consensus framework”, “OHDSI’s recommended practice”, “OHDSI’s process” and a dozen other terms each time you refer to it is not a good idea.
I think we have an opportunity to declare that nobody has figured out true validation: the source record stuff is one-sided and so arduous that it kills many of the advantages of RWE, while the probabilistic stuff is just an indication, not a fact. That means, work is not done for the regulators (“source-record validate” is not the solution) and us (we need to figure out sensitivity and specificity), or we need methods taking the measurement error into account.
Of course, my main bone of contention, as you know, is the point of the “clinical description”. It is both undefined as well as useless when folks just paste medical textbook chapters or GPT text. What we really talking about is research purpose and how it aligns with the artifacts relevant to the clinic. I give you an example. A couple of Phebruaries ago, we had the phenotype of “Parkinson’s disease”, and the question came up if drug-induced parkinsonism should be part of it or not. Obviously, if you want to study, say, treatments modulating neurodegenerative pathogenetic processes, it should not be included. If, however, you are studying symptomatic treatments or disease complications it very much should. Of course, you could make any of this part of the “clinical description” and ignore the suboptimal term, but right now it is not specified at all.
The other elephant on the table may be introduced as well: That phenotypes not only match healthcare data to research purposes, which is tricky even if the data were perfect, it also tries to deal with the inherent data quality issues (e.g. the infamous two diabetes diagnosis codes within a month to improve on specificity). Currently, we have nothing in the framework which specifies and validates these, and neither do the guidelines talk about it.