Phenotype Phebruary 2023
In 2022, we started many discussions related to phenotyping.
- Many perspectives on phenotyping were discussed. Some we agreed on, others are still debated.
- We started wanting a structured process for phenotype development and evaluation and so we experimented on many ways.
- Collaborators started to systematically use our OHDSI tools (ATLAS, CapR, PHOEBE, APHRODITE, CohortDiagnostics, PheValuator,…)
- Over time a pattern evolved that started looking like good practices.
- We built out the OHDSI Phenotype Library. We outlined a peer review process for content to get into the library, and specified what and how cohort definitions could get in, and how we retrieve and use them in our research.
Phenotype Phebruary 2022 gave us so many new ideas on good practices that matured over the year. It is time to test if those good practices are really best practices – can we harden them further in 2023? If we all focus the community effort to collaboratively review, evaluate and build a number of phenotypes together using the insights we learnt last year, we debate the ideas behind these practices – we may move us closer towards having a resource that can support all of our community analysis activities.
So, we have Phenotpye Phebrurary 2023:
For week 1:
Inputs: You will be given
- A target Phenotype
- A clinical description
- Prior work (literature review)
- One or More cohort definitions
- Output from OHDSI tools (CohortDiagnostics/PheValuator) on the definition executed on one or more datasource
- A written summary and interpretation of findings by an OHDSI collaborator
Deliverables: complete a peer review to these inputs and provide recommendation to
- accept the phenotype definition
- Modify the phenotype definition
- Reject the phenotype definition
Target: 4 phenotypes will be posted. One every day in the first week of February.
Debate topic (to take place at the OHDSI forums): What is peer review? What is the value of peer review? What should be in peer review? Who should do the peer review?
Looking for volunteers: Do you want to be part of Week 1? We are looking for collaborators to perform peer review existing phenotype definitions and evaluation summaries. A successful collaborator will think critically about what it entails to review another collaborator cohort definition and provide peer review judgments on the work. Collaborators will receive 1:1 assistance from organizers on their activities. No prior knowledge or skills required. Clinical understanding of the phenotype is preferred.
For week 2:
Inputs: You will be given
- A target Phenotype
- A clinical description
- Prior work (literature review)
- One or More cohort definitions
- Output from OHDSI tools (CohortDiagnostics/PheValuator) on the definition executed on one or more datasource
A written summary and interpretation of findings by an OHDSI collaborator
Deliverable 1: a written evaluation report that summarizes your findings
Deliverable 2: complete a peer review to these inputs and provide recommendation to
- accept the phenotype definition
- Modify the phenotype definition
- Reject the phenotype definition
Target: 3 phenotypes will be posted, spread across the second week of February. Activities may be started earlier.
Debate topic (to take place at the OHDSI forums): What is evaluation and why should we empirically evaluate cohort definitions? What to look for and perform an evaluation? What are the operating characteristics of a cohort definition? How helpful are the tools (Cohort Diagnostics, PheValuator, Cohort Explorer)? What is a gold standard and do we truly need it?
Looking for volunteers: Do you want to be part of Week 2? We are looking for collaborators to perform the two deliverables. For Deliverable 1: You will be a) Interpret cohort diagnostics and/or phevaluator output, b) Modify phenotype definition when required, c) Author a summary of the evaluation. For Deliverable 2: Collaborators are needed to complete the peer review.
For week 3: You will be given
- A target Phenotype
- A clinical description
- Prior work (literature review)
- One or More cohort definitions
Output from OHDSI tools (CohortDiagnostics/PheValuator) on the definition executed on one or more datasourceA written summary and interpretation of findings by an OHDSI collaborator
Deliverable 1: execute OHDSI tools (CohortDiagnostics/PheValuator) on the definition on the OHDSI network and get output. Make results publicly available e.g., shiny on data.ohdsi.org
Deliverable 2: a written evaluation report that summarizes your findings
Deliverable 3: complete a peer review to these inputs and provide recommendation to
- accept the phenotype definition
- Modify the phenotype definition
- Reject the phenotype definition
Target: 2 phenotypes will be posted, spread across the third week of February. Activities may be started earlier.
Debate topic (to take place at the OHDSI forums): What makes cohort definitions reusable? Do we customize phenotype development to data and/or analytical use case (study) or, is phenotyping an independent activity? What terminology should we use when we describe operating characteristics of a cohort definitions (e.g., sensitive cohort, specific cohort). What is the value of OHDSI PL and what should be in it?
Looking for volunteers: Do you want to be part of Week 3? We are looking for collaborators to perform the three deliverables. For Deliverable 1: Data partners with access to data on OMOP CDM and willing to run Cohort Diagnostics and submit results. For Deliverable 2: You will be a) Interpret cohort diagnostics and/or phevaluator output, b) Modify phenotype definition when required, c) Author a summary of the evaluation. For Deliverable 3: Collaborators are needed to complete the peer review.
For week 4: You will be given
- A target Phenotype
- A clinical description
- Prior work (literature review)
One or More cohort definitionsOutput from OHDSI tools (CohortDiagnostics/PheValuator) on the definition executed on one or more datasourceA written summary and interpretation of findings by an OHDSI collaborator
Deliverable 1: collaborative develop one or more cohort definitions for the target. Seek clarification of the clinical idea and make changes as needed. Use material from literature review. Build cohort definitions that may be run on OMOP CDM that may be either rule based on probabilistic but the output should be cohort.
Deliverable 2: execute OHDSI tools (CohortDiagnostics/PheValuator) on the definition on the OHDSI network and get output. Make results publicly available e.g., shiny on data.ohdsi.org
Deliverable 3: a written evaluation report that summarizes your findings
Deliverable 4: complete a peer review to these inputs and provide recommendation to
- accept the phenotype definition
- Modify the phenotype definition
- Reject the phenotype definition
Target: 1 phenotype will be posted, in the fourth week of February. Activities may be started earlier.
Debate topic (to take place at the OHDSI forums): The role of probabilistic modelling in phenotype development and evaluation. Can we sustain this level of effort for rule-based phenotyping, does it scale?
Week 5: Synthesis
Debate topic (to take place at the OHDSI forums): Is phenotyping a scientific work that is worthy of scientific dissemination via publication? How was comparing to Gold standard useful, feasible, limitations
