PoC: compiling ClinicalTrials.gov eligibility prose into a gate-verified Circe cohort, with refusal as a first-class output

Researchers
, ,
1

I built a proof of concept for the eligibility-to-cohort authoring step: an LLM reads the verbatim registered criteria of a trial at build time and emits a standard Circe cohort definition, committed only after passing four automated gates. The runtime is unmodified OHDSI tooling (CirceR, SqlRender, DatabaseConnector), no model in the execution path. Where the prose admits no single determinate encoding over the CDM, the criterion compiles to AMBIGUOUS with a rationale, never to a guess.

NCT03667300 (evogliptin, T2DM with renal insufficiency) is the example: 16 criteria, of which 10 compile determinate and 6 refuse. The gates: valid Circe structure with every concept_id resolving against a pinned vocabulary subset; a bijection between source sentence ids and manifest entries, so no criterion is silently dropped or invented; execution against 29 synthetic patients with expected labels, including pinned boundary cases (eGFR 30 vs 29.9, HbA1c 7.0 vs 6.9); and byte-identical recompile.

The behavior I think is worth a look is the refusal on E6, “strong CYP3A4 inhibitors or inducers”. A patient on itraconazole would be correctly excluded by a guess, itraconazole is a strong CYP3A4 inhibitor, and the tool refuses anyway, because there is no vocabulary-native concept set for the DDI-strength category, so encoding it means inventing membership. The refusal ledger in docs/AMBIGUOUS.md lists all six with what would make each determinate.

Honest state: one trial, synthetic fixtures with author-asserted labels, not clinically adjudicated, one Athena snapshot pinned. It is a methodology PoC, not a recruitment tool and not a benchmark. Chia and n2c2 are treated as evaluation-only and never enter the compile input. Closest prior work is Criteria2Query; the difference attempted here is the committed, fixture-verified artifact and the tested refusal class rather than best-effort extraction.

Gates 1, 2 and the determinism check run with Python only, no API key: clone, uv venv && uv pip install -e '.[dev]', python -m pytest -q. Full execution path runs in the devcontainer.

Repo: GitHub - compiled-ai-labs/trial-eligibility-compiler: An LLM compiles verbatim ClinicalTrials.gov eligibility criteria into a deterministic, gate-verified OHDSI Circe cohort — at compile time only, no model in the runtime. Indeterminate criteria compile to AMBIGUOUS, never a guess. A Compiled AI reference implementation. · GitHub

The question for people who author cohort definitions by hand: do the six refusals land where you would expect? If an experienced phenotyper looks at the AMBIGUOUS list and says “E5 is actually encodable, here’s how” or “you missed that I6 is also indeterminate”, that is the most useful thing I could learn from this community.

1 Like
2

@boristep:

Looks like a clean tool people could trust. Because of your cautious approach to err on the side of precision, rather than trying to cover every single criterion.

But you may want to state the purpose or use cse of the tool. If you want to use it for patient recruitment, ignoring criteria which are inherently based on local human assessement is good enough – all you need to produce is a rather inclusive list of potential candidates. If you want to use the tool for reproducing the clinical trial in the Real World, you need to more aggressively interpret criteria that are not the usual yada yada yada type (“patient understands consent”), but can materially change the underlying biological question. For example, which drugs inhibit or induce CYP3A4 is known, and you’d need to create that reference list from public knowledge, which is a core strength of the foundational LLMs. Which of these drugs are strong is an interpretation, but as long as you have a rationale you can justify your pick – at the end of the day the recruiting physician does exactly that.

Good luck.

3

Thank you, this is exactly the critique I hoped for.

You are right that I left the purpose implicit, and it changes the refusal threshold. For a recruitment candidate list, an inclusive set with human-assessment criteria dropped is the goal. For reproducing the trial in real-world data, refusing a criterion that materially changes the biological question is the wrong default. I will state the intended use in the repo. This PoC is aimed at the reproduction case, which is the stricter one.

On E6 you have found a real seam in my reasoning. I conflated two things: “no vocabulary-native concept set exists” and “the classification requires interpretation.” The first is a genuine gap. The second, which is what CYP3A4 strength actually is, is not a refusal case if I carry a documented rationale, and that is the same discipline the rest of the tool already applies to its artifacts. Building the inhibitor and inducer reference list from public knowledge and pinning the strength picks with a justification is closer to what the recruiting physician does than a blank refusal. That would move E6 out of AMBIGUOUS and into an interpreted concept set with provenance.

The interesting question this raises: the difference between “refuse” and “interpret with rationale” is a policy line, not a technical one. Where a hand phenotyper draws it is what I would most like to understand.

4

Makes total sense.

But even the reproduction use case is not that straightforward: Do you want to (i) get EXACTLY the same patient population the clinical trialists seem to have had in mind, or (ii) do you want to answer the same biological question, but with the actual real-world patients? The difference is significant: the former is trying to create the smallest possible and cleanest population that gives you the maximum effect size, with all distractors and confounders taken out. The latter is going for generalization of an effect in the real world the clinical trial people love to take for granted.

The CYP3A4 critierion is a classic example for this. Why do they exclude BOTH blockers and inducers of a liver enzyme? Because it metabolizes many drugs, and blockers will increase the effective dose of the trial drug and the inducers will decrease it. Either way it muddies the water, and muddier water requires larger sample size or subpopulation analysis, which usually also requires larger sample size. Larger sample size = higher trial cost. So, they kick them out. But in the real world, many of the drugs falling into that category are commonly taken, and in order to measure drug effect in the real world you should really not exclude them.

The solution requires justifying the purpose of each criterion, rather than just accept them as if the Lord handed them down to us.

And then there is another elephant on the table: Criteria only real-world studies require because the real-world data are not exact reproductions of the biology, but artifacts of running the healthcare system. For example, the notorious hospitalization as a surrogate for severity, the re-diagnosing through addition of procedures and lab tests because we don’t trust diagnostic codes and the requirement for the same fact more than once within a time period for the same reason.

Are you sure you want get bogged down in that swamp? :slight_smile:

5

I do want to get into that swamp, but as an engineer, not as someone pretending to judge the biology.

I am not a doctor or a biologist. The tool should never decide any of what you described. What I saw in your two replies is a community doing hard work under contradictory criteria, where the correct encoding depends on purpose, on the difference between cleanest effect size and real-world generalization, and on how much of a criterion is real biology versus an artifact of how the healthcare system records data. Those are decisions specialists must own.

Where I think an engineer can help is the layer under those decisions: surfacing each one explicitly, carrying the context needed to make it, and recording the rationale once a specialist has decided. Not a decision-support system, there are plenty already. An instrument that makes the ambiguity visible and auditable instead of hidden in a guess. What I am trying to learn is whether the compile-time, gate-verified, refuse-or-flag shape of this PoC is actually useful for that, or whether it just relocates the same work.

Your reaction tells me it is at least worth pursuing, and that is what I most wanted to know. So I would like to understand the real pain points from people who do this by hand, and to contribute as an engineer, in the same volunteer way the OHDSI community already works. If there is a working group, a set of real criteria, or a person I should talk to, point me at it. I am glad to put engineering effort in, and if this grows into something that needs proper backing, I am open to that conversation too.

6

I totally understand this approach, and have empathy.

But: I believe we need to overcome the distinction between the engineering solution and expert contribution. Reasons: (i) the latter simply doesn’t exist, at least not in the amount needed to create scalable and reproducable solutions (most RWE people are not medical professionals with in-depth knowledge of both disease biology and institutional artifacts), and (ii) the “expert” stuff is not all that complicated, and solutions can be engineered as well. We just have to tackle it.

For example, in the CYP3A4 situation can be systematically solved by a smart tool. You simply ask “How is the study drug metabolized, and is the outcome sensitive to slight dose deviations if metabolization is impaired or induced?”. ChatGPT knows the answer to that. If it is “yes”, the follow-on question is “are there conditions or other drugs that affect metabolization?” ChatGPT knows the answer to that. And then you drop an exclusion criterion.

Now, where do you get these questions from? That, again, only an expert can provide, correct? Wrong! I believe there may be a few dozen of these questions, and that is it. The community can collect them over time, and a cool developer like @boristep can turn them into the heuristic of a tool or agent. That solution will be a lot better than whatever we have (haven’t, really) today.

Feel ambitious? :slight_smile: