Concept set critic for ATLAS/OMOP phenotypes — looking for feedback

Developers
, ,
1

Hi everyone! I’m working on a validation-focused “critic” for OMOP concept sets and would love feedback from the OHDSI community.

The tool is meant to review an existing ATLAS / Phenotype Library concept set rather than generate one from scratch. Given a concept set plus a short phenotype intent description, it produces an audit report with evidence-backed findings, such as potentially over-broad descendants from includeDescendants, suspicious mappings from includeMapped, missing candidate concepts, possible exclusions, and aggregate CDM prevalence/co-occurrence signals for human review.

A key design constraint is that the LLM does not invent concept IDs. It only judges candidates already retrieved from OMOP vocabulary structure, PHOEBE-style relatedness, or CDM-derived aggregate evidence. Uncertain signals stay review-only rather than being automatically applied.

My main questions are:

  1. Would this kind of audit report fit anywhere in existing OHDSI phenotype development or review workflows?
  2. Which finding types would domain experts be most likely to trust, ignore, or want presented differently?
  3. Are there existing OHDSI tools, conventions, or evaluation practices I should align with more closely?

I’m especially interested in feedback on whether this framing is useful: not “LLM-generated phenotypes,” but human-in-the-loop concept set review with provenance for each finding.

2
  1. A feedback loop that improves reliability and reproducibility of Phenotype Development and Evaluation process is useful.
  2. In OHDSI we value reproducibility and reliability. We need empirical evidence that the tool enables that.
  3. Yes. GitHub - OHDSI/Phenelope: An R package for creating concept sets using large language models. · GitHub ; Jared presented another framework at Phenotype Development and Evaluation workgroup meeting last month.
3

Thanks, this is really helpful, and the pointers are exactly what I needed.

One clarification on where my tool sits: unlike Phenelope and Jared’s framework, it doesn’t generate concept sets. It audits an existing one — flagging over-broad includes, missing descendants, exclusion candidates, and redundant seeds — so it sits downstream of generation as a QA/review layer. It would compose with Phenelope rather than compete.

On empirical evidence, I’ve started collecting it:

  • Reproducibility: the audit backbone is deterministic (same set in → same findings out, no LLM). The optional LLM refine loop I’ve measured at ±1 member run-to-run, root-caused, with a fix to make it fully reproducible.
  • Benchmark: run across 15 concept sets, 4 OMOP domains, and 6+ therapeutic areas (17 → 54k members) — converges at every scale, domain guards hold, cost tracked per phenotype.

I’ll dig into Phenelope as well. Would these results be useful to bring to the Phenotype Development & Evaluation workgroup?

4

Yes. You are welcome to present of course! Please send me a direct message and let us schedule a presentation.

5

Thank you, I’ll send a direct message shortly!