Regarding
I agree that the term phenotype can confuse rather than clarify how we think and communicate about cohort definitions. Unfortunately there is too much functional similarity between this work and the strategies for linking biologic causes to their clinical expression as captured in health care data. It’s essentially the same practice being conducted for slightly different purposes. On the bright side, I think it’s only a matter of time before parts of those strategies like the ontologies (e.g. the Human Phenotype Ontology) and methods (e.g. semantic similarity analyses) come to OMOP and OHDSI and help us build better cohort definitions. @Juan_Banda’s great work is bringing us closer by the day. And once that day arrives, there will be no hope (and little reason) for using separate terms.
Regarding
I agree that’s needed and very important. I think there are at least four approaches available or currently being developed in OHDSI that can let us assess performance relative to a clear definition at some points in time for many, though not all, conditions.
-
Clear definitions of disease can be derived from sources in EHRs that are considered definitive such as pathology reports. At least in some cases these could be used as a gold standards to assess the performance of definitions restricted to Dx, Px, & Rx codes, etc when all are available on the same patients. Analogously, some lab values or imaging results are considered definitive for some conditions.
-
When tumor registry data are available on patients and fused with patients’ EHR data in a single OMOP instance, the tumor registry provides a clear definition of the disease and can function as a gold standard to assess performance of definitions that only use non-tumor registry sources.
-
Bringing clinical trial data into OMOP and fusing it with data on that same patients from an EHR/claims/registry data source will allow this same strategy for whichever diseases are definitively ascertained in the trial.
-
The capture of clinical expert’s judgments about case ascertainment by experts using Trey Schneider’s awesome Annotation tool can support the same gold standard performance assessment strategy.
In all 4 cases, I think we should develop standards for recording which definitions are considered definitive and why in the metadata schema and ontologies being developed in the Metadata WG.
These gold standards won’t always answer when a disease state begins. But they will allow performance to be assessed relative to a definitive assessment at a given point in time. I think that’s good enough. Doing much better than that might require near omniscience.
Though only some sites can do this work, comparison to a gold standard is of foundational importance in assessing performance. So, I think developing and exploiting these strategies should be a high priority. As we get better at assessing how performance in one data source is likely to extrapolate to another, sites that can’t do gold standard validation of cohort definitions in their own data will increasingly benefit from knowledge of performance at sites that can.