Open Challenge to AI Innovators: Participate in the OHDSI 2025 “Collaborative Intelligence” Scientific Evaluation
Dear OHDSI Community and Innovators in Clinical Informatics,
Following up on the initial ‘Call for Collaboration’ and the detailed experimental design (the “Run-of-Show”) posted previously, we are now actively recruiting for the “AI Arm” of this experiment.
The proposed OHDSI 2025 Symposium session, "Collaborative Intelligence: Humans and AI in Concept Set Development," is a "Minds Meet Machines" challenge designed as a rigorous, hypothesis-driven scientific study. The scientific rigor of this evaluation depends on comparing expert human curation against the most advanced, distinct (multiple) AI pipelines available.
To ensure this is a world-class, scientific evaluation of Humans versus AI in clinical modeling, we must actively seek out and challenge the leaders in this space.
We have conducted an extensive search spanning academic literature (PubMed, preprints, informatics journals), conference proceedings, open-source repositories (GitHub), and proprietary announcements. Our objective was to identify the academic teams, open-source projects, and industry leaders who are actively developing methodologies for AI-assisted concept set development, value set management, and clinical coding (including OMOP, ICD, HPO, and others).
The organizations listed below are the identifier innovators in our systematic search. If you have not been identified, please reach out to us now and you will be invited.
If your organization is listed, it is because you have publicly claimed innovation in this domain. To contact you, we are posting this here, and we are emailing you directly.
We invite you to bring your solutions and expertise to this open collaboration. We challenge you to validate your methodologies within our standardized, blinded, randomized comparative framework using the ~28 defined phenotype conditions detailed in the experimental protocol (see the first post in this thread).
To the Innovators Listed: Are you up to the challenge?
To the OHDSI Community: The success of this collaboration relies on bringing these innovations together. If you collaborate with these teams or individuals, we encourage you to facilitate an introduction or forward this invitation.
Below is the comprehensive list of identified innovators and their publicly available contact information, compiled entirely through public domain searches.
OHDSI 2025 “AI Arm” Challenge: Outreach Tracker
| Name | Key Person(s) / Public Contact | Why You (Focus/Innovation) | Link(s) | Response |
|---|---|---|---|---|
| Academic & Research Initiatives | ||||
| Columbia University DBMI | Chunhua Weng, PhD (chunhua@columbia.edu ); George Hripcsak, MD; Anna Ostropolets, MD, PhD anna.ostropolets@columbia.edu | Criteria2Query (C2Q) 3.0 (NLP to OMOP SQL); LLM for concept set curation (refining PHOEBE); Automated taxonomy learning. | Columbia DBMI | |
| Vanderbilt University Medical Center (VUMC) DBMI | Wei-Qi Wei, MD, PhD: wei-qi.wei@vumc.org ; Chao Yan, PhD: chao.yan.1@vumc.org | Evaluation of LLMs (GPT-4, Claude) for generating executable phenotyping algorithms (SQL queries) adhering to a CDM. | Vanderbilt DBMI; Large Language Models Facilitate the Generation of Electronic Health Record Phenotyping Algorithms | |
| Stanford University (AIMI / Shah Lab) | Nigam H. Shah, MBBS, PhD (nigam@stanford.edu ) | Extensive research on LLM evaluation in healthcare, including automated billing code assignment and AI fairness frameworks. | The Stanford Center for Artificial Intelligence in Medicine and Imaging (AIMI) | |
| UTHealth Houston & Mayo Clinic (Agentic MCP) | Hua Xu, PhD: Hua.Xu@uth.tmc.edu ; Hongfang Liu, PhD: Hongfang.Liu@uth.tmc.edu | Development of the Agentic Model Context Protocol (MCP) framework for zero-training, hallucination-preventive OMOP mapping using LLMs. | An Agentic Model Context Protocol Framework for Medical Concept Standardization | |
| King’s College London (KCL) / Alan Turing Institute (MedCAT Team) | Prof. Richard Dobson: richard.j.dobson@kcl.ac.uk | Developers of MedCAT/CogStack; NLP for extracting/linking clinical concepts to UMLS/SNOMED CT; Integrating Transformer models. | Medical Concept Annotation Tool | |
| UCSF - Bakar Computational Health Sciences Institute | Vivek Rudrapatna, MD, PhD: Vivek.Rudrapatna@ucsf.edu | Utilizing LLMs and NLP to transform unstructured narratives into structured data, including OMOP mapping and knowledge graphs (SPOKE). | UCSF Bakar Computational Health Sciences Institute | |
| Mass General Brigham (MGB) Research Team | Heekyong Park, PhD (Corresponding Author): hpark25@mgb.org | Utilizing LLMs + RAG to improve phenotyping accuracy from unstructured EHR data, comparing against traditional ICD code methods. | A Comprehensive Evaluation of LLM Phenotyping Using Retrieval-Augmented Generation (RAG): Insights for RAG Optimization | |
| Mount Sinai Health System (MSHS) | Eyal Klang, MD (Corresponding Author): eyal.klang@mountsinai.org | Evaluation of RAG-enhanced LLMs (GPT-4, Llama-3.1) for automating Emergency Department ICD-10-CM coding vs. human coders. | Assessing Retrieval-Augmented Large Language Model Performance in Emergency Department ICD-10-CM Coding Compared to Human Coders | |
| Yale School of Medicine | rohan.khera@yale.edu ; Samah Fodeh, PhD (Corresponding Author): samah.fodeh@yale.edu | Novel Sentence Transformer-based NLP approaches for mapping EHR data (medications) to the OMOP CDM. | A Novel Sentence Transformer-based Natural Language Processing Approach for Schema Mapping of Electronic Health Records to the OMOP Common Data Model | |
| CLH (Code Like Humans) | Univ. of Cambridge/Microsoft Research Authors andreas@motzfeldt.dk ; Yu-Neng Chuang (Corresponding Author): yc577@cam.ac.uk | LLM-based agentic framework designed to automate medical coding (ICD-10) by mirroring human processes. | Code Like Humans: A Multi-Agent Solution for Medical Coding | |
| MedCodER | Univ. of Illinois Urbana-Champaign Authors sanmitra1@gmail.com ; Y. Zhang & J. Gao (Corresponding Authors): {yuz9, jgao8}@illinois.edu | Generative AI framework for automatic medical coding (ICD code prediction) using extraction, retrieval, and re-ranking. | MedCodER: A Generative AI Assistant for Medical Coding | |
| Erasmus MC - Medical Informatics | Peter Rijnbeek, PhD (p.rijnbeek@erasmusmc.nl) | Exploring NLP and LLMs for extracting concepts from clinical text (including non-English languages) for phenotyping. | Erasmus MC - Medical Informatics | |
| The Jackson Laboratory & Charité Berlin | Peter N. Robinson, MD, MSc (Peter.Robinson@jax.org) | Development of Human Phenotype Ontology (HPO); Computational deep phenotyping and ontology-based data models (Phenopackets). | The Jackson Laboratory & Charité Berlin | |
| KOMAP | University of Florida and NVIDIA Authors tcai@hsph.harvard.edu | Knowledge-Driven Online Multimodal Automated Phenotyping; Uses knowledge graph embeddings on EHR concepts to generate feature lists. | Knowledge-Driven Online Multimodal Automated Phenotyping System | |
| Open-Source Tools & Consortia | ||||
| The Monarch Initiative | Melissa Haendel, PhD; Chris Mungall, PhD info@monarchinitiative.org | International consortium for deep phenotyping; NLP/ML to extract phenotypic information and standardize using ontologies (HPO). | Monarch Initiative | |
| Llettuce | University of Manchester Authors g.figueredo@nottingham.ac.uk R.A.C. and G.D. (Corresponding Authors): {rebecca.croft, goran.davidovic}@manchester.ac.uk |
Open-source tool using local LLMs, semantic search, and fuzzy matching for converting medical terms into the OMOP standard vocabulary. | Llettuce: An Open Source Natural Language Processing Tool for the Translation of Medical Terms into Uniform Clinical Encoding | |
| RAG-HPO | Baylor/Texas Children’s Hospital Authors Jennifer.Posey@bcm.edu | Python tool using RAG to enhance LLM accuracy in assigning Human Phenotype Ontology (HPO) terms from medical free-text. | Improving Automated Deep Phenotyping Through Large Language Models Using Retrieval Augmented Generation | |
| GCAF (Generalised Codelist Automation Framework) | A. Aslam (Corresponding Author): a.aslam@sheffield.ac.uk | Framework and Python repository for automating the development of clinical codelists (Readcodes, SNOMED). | An automation framework for clinical codelist development validated with UK data from patients with multiple long-term conditions | |
| CLAMP (Clinical Language Annotation, Modeling, and Processing) | Hua Xu, PhD (Hua.Xu@uth.tmc.edu ) | Comprehensive clinical NLP toolkit; Tools for entity recognition and mapping clinical concepts to UMLS/SNOMED CT. | CLAMP Clinical Language Annotation, Modeling, and Processing Toolkit | |
| OHNLP/omop_mcp | jaerongahn@gmail.com | Open Health NLP; Open-source Model Context Protocol (MCP) server for mapping clinical terminology to OMOP concepts using LLMs. | GitHub omop_mcp | |
| pyomop | bell@nuchange.ca | Python library for OMOP CDM; Includes LLM-based natural language queries and LLM-agent assisted FHIR to OMOP mapping. | GitHub pyomop ; Vibe Coding FHIR to OMOP - Bell Eapen MD, PhD. | |
| MedRAG / MIRAGE Benchmark | Univ. at Buffalo, NCBI/NLM/NIH Authors | Toolkit (MedRAG) and benchmark (MIRAGE) for the systematic evaluation of RAG systems in the medical domain. | Benchmarking Retrieval-Augmented Generation for Medicine | |
| OHDSI Community Efforts | ||||
| OHDSI Generative AI Workgroup | OHDSI Community Members | Official workgroup dedicated to the application of generative AI for RWE, including automated taxonomy learning and concept sets. | Large Language Models Can Enhance OHDSI Evidence Generation Mission | |
| CHIMERA | OHDSI Contributors | Initiative focused on automatic concept set creation and mapping to standard OMOP codes within the OHDSI ATLAS platform. | CHIMERA: Automatic Concept Set Creation and Mapping to Standard OMOP Codes in ATLAS | |
| LLM-Powered OMOP Conversion Tool | Artem Naumenko (OHDSI: ents) daniel@hyperunison.com | Community initiative utilizing an LLM-first approach for automated structural and semantic mapping to accelerate OMOP conversion. | Accelerating OMOP Conversion with free LLM-Powered Tool – Looking for Test Users | |
| Proprietary Software & Industry | ||||
| RWE & Data Harmonization Platforms | ||||
| IQVIA | Contact Page | AI solutions to accelerate OMOP conversions and AI-driven analytics for RWE generation. | IQVIA HealthGrade AI | |
| Evidentli | info@evidentli.com | Piano and Auto-Mapper tools; Uses ML/NLP to automate the standardization of medical concepts for OHDSI CDM transformation. | Evidentli | |
| OM1 | info@om1.com | PhenOM AI Platform; AI-powered digital phenotyping to create digital phenotypic “fingerprints” for cohort identification. | OM1 | |
| Nference | Google Search | nferX AI platform; Advanced NLP to normalize data and map concepts to standardized ontologies for RWE generation. | nference | |
| Truveta | Google Search | “Truveta Language Model” (TLM) to clean, harmonize, and normalize disparate EHR data, including extensive concept mapping. | Truveta | |
| Terminology & NLP Providers | ||||
| IMO Health | CustomerSupport@imohealth.com | LLMs for automated medical coding (ICD-10, CPT, SNOMED CT) and “smarter value set management,” enhanced with RAG and proprietary terminology. | IMO Health | |
| John Snow Labs | info@johnsnowlabs.com | Healthcare NLP & LLM library; Healthcare-specific LLMs to extract entities and map them to standardized terminologies (RxNorm, ICD, SNOMED, HPO). | John Snow Labs | |
| Autonomous & AI-Assisted Coding | ||||
| Solventum (Formerly 3M HIS) | Helps.us@solventum.com | 360 Encompass™ system; AI-powered autonomous and computer-assisted coding solutions. | Solventum | |
| Fathom Health | hi@fathomhealth.com | Deep learning and LLMs to automate medical coding and auditing (E/M, CPT/HCPCS, ICD). | Fathom Health | |
| BUDDI.AI | Google Search | CODING.AI; Deep learning platform to automate medical coding (CPT, ICD 10). | BUDDI.AI | |
| Nym Health | Contact Nym | Trusted Medical Coding & Billing Software | Autonomous medical coding using “Clinical Language Understanding (CLU)” to translate notes into billing codes (ICD-10, CPT). | Nym Health | |
| CodaMetrix | hello@codametrix.com | AI-Powered Contextual Coding Automation Platform; Uses NLP and a “knowledge graph” to improve coding quality. | CodaMetrix | |
| Maverick Medical AI | info@maverick-ai.com | Autonomous medical coding platform powered by deep learning, aiming for high “Direct-to-Bill” rate. | Maverick Medical AI | |
| MediCodio | info@medicodio.ai | AI-powered coding automation; Hybrid strategy using LLMs/AI with SNOMED CT for clinical interpretation, then converting to ICD-10. | Medicodio | |
| AGS Health | https://www.agshealth.com/contact-us/ | Intelligent Authorization® Autonomous Coding; NLP/ML to automate ICD-10-CM, PCS, CPT, and E&M code assignment. | AGS Health | |
| AutoICD | info@autoicdapi.com | AI-powered Clinical Coding API; NLP to process unstructured texts and generate structured data in SNOMED-CT and ICD10. | AutoICD | |
| MediMobile (Genesis) | support@MediMobile.com | Autonomous medical coding powered by AI (Genesis); Automates CPT & ICD-10 coding in real-time from documentation. | MediMobile | |
| Ambient AI & Documentation | ||||
| Abridge | Connect with AI Healthcare Experts | Contact Abridge | Ambient AI for medical conversations; Real-time mapping of spoken clinical concepts to ICD-10, SNOMED, CPT. | Abridge | |
| Suki AI | support@suki.ai | AI-powered voice solutions; Automatically generates suggested ICD-10 and CPT codes based on captured clinical context. | Suki AI | |
| Prometheus | sarah_seager@epam.com | Prometheus is a powerful and versatile platform that can enable collaborative research and data analytics in the healthcare industry. Its comprehensive features, compatibility with industry standards, and focus on security and collaboration make it a valuable tool for researchers and healthcare organizations seeking to leverage the power of data to improve patient outcomes. | https://solutionshub.epam.com/solution/prometheus |
Call to Action for the AI Arm
This experiment is intended to generate high-quality, peer-reviewable evidence regarding the integration of AI into the phenotyping lifecycle.
If your organization is interested in submitting your AI pipeline for this evaluation, please REPLY to this forum thread or contact me directly rao@ohdsi.org .
As noted in the experimental stipulations:
“To ensure reliability and fairness, the AI-generated concept sets have to be generated prior to the symposium using the exact same standardized inputs provided to the human teams.”
“AI Methodology Transparency: The AI prompt and system settings (e.g., agent, version, temperature) used for the pre-generated AI concept sets will be documented.” However, you do not have to reveal your proprietary secrets. Also this is not an opportunity to market yourself.
We will coordinate with participating teams on the next steps for this crucial pre-symposium generation phase.
Sincerely,
Gowtham Rao
On behalf of the Phenotype Development and Evaluation Workgroup Organizing Leads & Contributors (Gowtham A Rao MD, PhD; Azza A Shoaibi PhD; Joel Swerdel PhD; Jack Murphy MPH, PhD)