Thanks @Gowtham_Rao for rekindling this conversation, and to @edburn for the valuable feedback. I would hope we can connect this thread with all of the other prior discussions that have taken place in our community for the past attempts, if for no other reason than to avoid rehashing old discussions. Here’s one of the prior threads as a reference that should be reviewed to orient the discussion.
I think this 2021 Gigascience paper, " Desiderata for the development of next-generation electronic health record phenotype libraries" offers useful food for thought, and would encourage others to give it a solid read. Perhaps a workgroup journal club that digs into the specific assertions would be helpful. I know some of the authors are in our community, can only find to tag @lrasmussen, but hope they could chime in and contribute to that discussion.
I’ll also note that, within the design of capR, @lee_evans and @mdlavallee92 , were very strategic in trying to separate out the components that made up a cohort definition (e.g. like conceptsets and inclusion criteria) in addition to allowing for their aggregation. So, a legitimate question to tackle (at some point) is whether our phenotype library is really a library of cohort definitions (with supporting metadata) or is a phenotype resource library containing various resource types that can be configured and built into cohort definitions (inclusive of cohort definitions themselves).
There are the technical implementation bits (conceptsets, cohort definitions), which I feel good we have a good current handle of within our community. There are the evaluation bits (such as, but not limited to, CohortDiagnostics and PheValuator) which are quite promising but still maturing in form and function. And then there’s the meta-data bits, and that’s where I struggle the most. It isn’t clear to me what is required, what is a desired nice-to-have, and what is distracting noise.
My current opinion: I think having a clinical description that states the intent of the phenotype target is very important to have and to connect with all elements used to implement that target. Otherwise, in absence of a description, it seems to be impossible to precisely evaluate the implementation, because you don’t know for sure what you were trying to accomplish. All too often, we assume we know the intent based only on a phenotype label (e.g. ‘Type 2 diabetes mellitus’ or ‘Delirium’ or ‘Hemorrhagic events’) only to realize that there is more nuance to it (e.g. Type 2: does that exclude T1DM and gestational diabetes, what does it mean to be ‘incident’; Delirium: does that include general confusional state, is it restricted to only events in hospital? Hemorrhagic events: is it bleeding-related hospitalizations, bleeding in hospitals, bleeding requiring health service utilization, or all bleeds all the time?)
If a definition has been evaluated, it would be useful to capture those insights that users learned from their evaluation. I don’t think simply providing results from CohortDiagnostics is sufficient, at least not at this current state where we don’t have objective criteria that are applied to the results to determine fitness-for-use. We have tried some ‘evaluation templates’, but I know not everyone is enamored with filling them in, so I don’t know exactly what the right solution is there.
@edburn raised the possibility of trying to understand the journey through phenotype development and documentation of choices along the way. I was discussing a similar topic with @jennareps as it relates to our analysis development and evaluation process: for reproducibility purposes, I think we need only have the final process fully specified and executable end-to-end (so I don’t need to know that I iterated a few times and stumbled before getting to the starting line, I only need to know the path from start to finish). but for purposes of transparency of process and education and awareness, seeing the full meandering journey through all versions could be informative…I just don’t know how practically realistic it is to expect others to provide this information.
As we go down this path again, I think it’s important to clarify the intent and target audience of the phenotype library: Personally, I’d say let’s not focus on how to put information in, but rather on how information within can be used by others to enable reliable evidence generation. If I could log into ATLAS or start up an R session with the HADES packages loaded, and know that I could confidently re-use a resource of community-evaluated cohorts as my building block inputs to my analyses (instead of having to start each study with denovo phenotype development), I’d be a much happier researcher.