I like Martijn’s plenary talk on using LLMs for case adjudication. The approach involved feeding patient-level narratives to the LLM, which then performed case-level adjudication to determine case or non-case scenarios. The adjudication performed by the LLM seemed good.
The narratives given to it were well-structured, encompassing details such as the patient’s age, gender, and other demographics, along with their presentation, assessment, treatment regimen, and the nature of the visit. The structuring process involved a critical step where an individual, either Patrick or Anna, crafted rules to filter out irrelevant conceptIds from the plethora of conceptIds a patient might have around the time of diagnosis. For instance, in a case concerning ‘Rheumatoid Arthritis,’ unrelated conceptIds like ‘ear pain,’ ‘sore throat’ were filtered out. This refined data was then handed over to the keeper for a mini feature extraction pertaining to the case under review. In essence, Patrick and Anna acted as the ‘experts,’ aiding the LLM in focusing on relevant information by eliminating the ‘noise.’
This process made me think - can we potentially automate the ‘expert’ role performed by Patrick and Anna using LLM? Can we then use it with CohortDiagnostics?
So I engineered this prompt
"Assume you are a specialist physician with expertise in the disease: Drug-Induced Liver Injury (DILI). Your task is to compile a markdown table that encompasses a comprehensive list of medical terms, including synonyms, associated with the disease. The table should have two columns: ‘Terms’ and ‘Clinical Domain’.
Instructions:
Under the ‘Terms’ column, list medical terms relevant to the disease, covering the categories below:
- Early symptoms of the disease
- Signs indicative of the disease
- Defining terms associated with the disease
- Relevant tests for diagnosing the disease
- Treatments for managing the disease
- Conditions that commonly co-occur with the disease
- Drug ingredients used in the treatment of the disease
- Visit settings where the disease is likely to be managed (e.g., inpatient, outpatient, doctor’s office).
In the ‘Clinical Domain’ column, categorize each term into one of the following: ‘Symptom’, ‘Sign’, ‘Defining Term’, ‘Test Name’, ‘Treatment Name’, ‘Co-occurring Condition’, ‘Visit Setting’.
Ensure your table is thorough and demonstrates your specialized knowledge in the disease."
it gave me
Terms Clinical Domain Jaundice Symptom Fatigue Symptom Abdominal pain Symptom Nausea Symptom Hepatomegaly Sign Elevated liver enzymes Sign Hyperbilirubinemia Sign Ascites Sign Hepatotoxicity Defining Term Idiosyncratic liver injury Defining Term Liver necrosis Defining Term Hepatocellular injury Defining Term Liver function tests (LFTs) Test Name Alanine aminotransferase (ALT) Test Name Aspartate aminotransferase (AST) Test Name Bilirubin levels Test Name Alkaline phosphatase levels Test Name Hepatic ultrasound Test Name Liver biopsy Test Name Discontinuation of offending drug Treatment Name Corticosteroids Treatment Name Ursodeoxycholic acid Treatment Name Liver transplant Treatment Name Alcoholic hepatitis Co-occurring Condition Viral hepatitis Co-occurring Condition Autoimmune hepatitis Co-occurring Condition Acetaminophen Drug Ingredient Amoxicillin-clavulanate Drug Ingredient Isoniazid Drug Ingredient Hospital (Inpatient) Visit Setting Liver clinic (Outpatient) Visit Setting Doctor’s office Visit Setting
That was nice. Now i went to CohortDiagnostics. I went to @Azza_Shoaibi work presented at ICPE 2023 presentation posted here https://data.ohdsi.org/icpe_2023_designated_medical_events/ (Cohort Diagnostics output screen shot). In the characterization diagnostics, i downloaded the spreadsheet and copied the top 50 features in the condition, procedure, observation, measurement domain for the time window 0, -1 to 30, +1 to 30 time windows.
"Based on the table created previously, and utilizing your medical expertise, categorize the following terms according to their clinical relevance to the Disease. The categories are:
Related: High confidence that the term reflects the clinical aspects of the disease.
Potentially Related: Probable clinical association with the disease.
Potentially Unrelated: Lesser likelihood of clinical association with the disease.
Unrelated: No clinical correlation with the disease.
Disqualifier: Clinical terms or conditions that, when present, significantly reduce the likelihood or rule out the presence of the disease.Present the results in a markdown table with ‘Terms’ and ‘Relationship’ as column headers. In the ‘Relationship’ column, specify one of the four categories: ‘Related’, ‘Potentially Related’, ‘Potentially Unrelated’, or ‘Unrelated’.
Terms: ‘Inflammatory disease of liver’, ‘Laboratory - General Classification’, ‘Laboratory - Chemistry’, ‘Pharmacy - General Classification’, ‘Laboratory - Hematology’, ‘Emergency Room - General Classification’, ‘EKG/ECG - General Classification’, ‘Jaundice’, ‘Essential hypertension’, ‘Laboratory - Bacteriology and Microbiology’, ‘Medical/Surgical Supplies and Devices - Sterile’, ‘Hepatic failure’, ‘Other Imaging Services - Ultrasound’, ‘Pharmacy - IV Solutions’, ‘Radiology - Diagnostic - General Classification’, ‘Laboratory - Urology’, ‘Radiology - Diagnostic - Chest X-Ray’, ‘Medical/Surgical Supplies and Devices - General Classification’, ‘Hyperlipidemia’, ‘Drugs Identification - Drugs Requiring Detailed Coding’, ‘Laboratory - Immunologu’, ‘History of event’, ‘Acute renal failure syndrome’, ‘Ground mileage, per statute mile’, ‘CT Scan - Body Scan’, ‘Respiratory Services - General Classification’, ‘Type 2 diabetes mellitus without complication’, ‘Anemia’, ‘Patient encounter procedure’, ‘Physical Therapy - General Classification’, ‘Physical Therapy - Evaluation Or Re-Evaluation’, ‘Abdominal pain’, ‘Anesthesia - General Classification’, ‘Operating Room Services - General Classification’, ‘IV Therapy - General classification’, ‘Sepsis’, ‘Pulmonary Function - General Classification’, ‘Blood Storage and Processing - General Classification’, ‘CT Scan - Head Scan’, ‘Ambulance service, advanced life support, emergency transport, level 1 (als 1 - emergency)’, ‘Acidosis’, ‘Recovery Room - General Classification’, ‘CT Scan - General Classification’, ‘Mental Health Substance Abuse Coverage Indicator’, ‘Dyspnea’, ‘Long-term current use of drug therapy’, ‘Chest pain’, ‘Chronic obstructive lung disease’, ‘Congestive heart failure’, ‘Ascites’, ‘Type 2 diabetes mellitus’, ‘Urinary tract infectious disease’, ‘Atrial fibrillation’, ‘Cirrhosis of liver’, ‘Malaise’, ‘Blood chemistry abnormal’, ‘Clinic - General Classification’, ‘Atherosclerosis of coronary artery without angina pectoris’, ‘Gastroesophageal reflux disease without esophagitis’, ‘Benign essential hypertension’, ‘Room & Board - Semi-private Two Bed (Medical or General) - General classification’, ‘Medical/Surgical Supplies and Devices - Other Implants’, ‘Occupational Therapy - General classification’, ‘Pleural effusion’, ‘Pure hypercholesterolemia’, ‘Low back pain’, ‘Other Diagnostic Services - Peripheral Vascular Lab’, ‘Long-term current use of anticoagulant’, ‘Laboratory Pathological - General Classification’, ‘Disease of liver’, ‘Thrombocytopenic disorder’, ‘Occupational Therapy - Evaluation or Re-evaluation’, ‘Cough’, ‘Operating Room Services - Minor Surgery’, ‘Medical/Surgical Supplies and Devices - Non Sterile’, ‘Nausea and vomiting’, ‘Hypokalemia’, ‘Pharmacy - Drugs incident to Radiology’, ‘Gastro Intestinal Services - General Classification’, ‘Heart failure’, ‘Hypo-osmolality and or hyponatremia’, ‘Cirrhosis - non-alcoholic’, ‘Asthenia’, ‘Hypothyroidism’, ‘Iron deficiency anemia’, ‘Echocardiology’, ‘Chronic kidney disease stage 3’, ‘Pharmacy - Other Pharmacy’, ‘Chronic kidney disease’, ‘Treatment/Observation Room - Treatment Room’, ‘Chronic hepatitis C’, ‘Dehydration’, ‘Treatment/Observation Room - Observation Room’, ‘Single liveborn born in hospital by vaginal delivery’, ‘Mixed hyperlipidemia’, ‘Obstructive sleep apnea syndrome’, ‘Laboratory Pathological - Histology’, ‘Fever’, ‘Room & Board - Semi-private Two Bed (Medical or General) - Medical/Surgical/Gyn’, ‘Atelectasis’, ‘Electrocardiogram abnormal’, ‘Muscle weakness’, ‘Newborn - Level I’, ‘Long-term current use of insulin’, ‘Abnormal findings on diagnostic imaging of lung’, ‘Diarrhea’, ‘Edema’, ‘Obesity’, ‘Skilled Nursing - Visit Charge’, ‘Cardiology - General Classification’, ‘Altered mental status’, ‘Low blood pressure’, ‘Disorder of brain’, ‘Acute hypoxemic respiratory failure’, ‘Leukocytosis’, ‘Ambulance service, basic life support, non-emergency transport, (bls)’, ‘Cardiomegaly’, ‘Obstruction of bile duct’, ‘Respiratory failure’, ‘Disorder of biliary tract’, ‘Alcoholic cirrhosis’, ‘Pneumonia’, ‘Septic shock’, ‘Gastrointestinal hemorrhage’, ‘Liver function tests abnormal’, ‘Portal hypertension’, ‘Hypoxemia’"
and i got
Terms Relationship Inflammatory disease of liver Related Laboratory - General Classification Potentially Related Laboratory - Chemistry Related Pharmacy - General Classification Potentially Related Laboratory - Hematology Potentially Related Emergency Room - General Classification Potentially Unrelated EKG/ECG - General Classification Potentially Unrelated Jaundice Related Essential hypertension Unrelated Laboratory - Bacteriology and Microbiology Potentially Unrelated Medical/Surgical Supplies and Devices - Sterile Potentially Unrelated Hepatic failure Related Other Imaging Services - Ultrasound Related Pharmacy - IV Solutions Potentially Related Radiology - Diagnostic - General Classification Potentially Related Laboratory - Urology Potentially Unrelated Radiology - Diagnostic - Chest X-Ray Potentially Unrelated Medical/Surgical Supplies and Devices - General Classification Potentially Unrelated Hyperlipidemia Unrelated Drugs Identification - Drugs Requiring Detailed Coding Related Laboratory - Immunology Potentially Related History of event Potentially Related Acute renal failure syndrome Potentially Related Ground mileage, per statute mile Unrelated CT Scan - Body Scan Potentially Related Respiratory Services - General Classification Potentially Unrelated Type 2 diabetes mellitus without complication Unrelated Anemia Potentially Related Patient encounter procedure Potentially Unrelated Physical Therapy - General Classification Unrelated Physical Therapy - Evaluation Or Re-Evaluation Unrelated Abdominal pain Related Anesthesia - General Classification Potentially Unrelated Operating Room Services - General Classification Potentially Unrelated IV Therapy - General classification Potentially Related Sepsis Potentially Related Pulmonary Function - General Classification Potentially Unrelated Blood Storage and Processing - General Classification Potentially Unrelated CT Scan - Head Scan Potentially Unrelated Ambulance service, advanced life support, emergency transport, level 1 (als 1 - emergency) Potentially Unrelated Acidosis Potentially Related Recovery Room - General Classification Potentially Unrelated CT Scan - General Classification Potentially Related Mental Health Substance Abuse Coverage Indicator Unrelated Dyspnea Potentially Unrelated Long-term current use of drug therapy Related Chest pain Potentially Unrelated Chronic obstructive lung disease Unrelated Congestive heart failure Unrelated Ascites Related Type 2 diabetes mellitus Unrelated Urinary tract infectious disease Unrelated Atrial fibrillation Unrelated Cirrhosis of liver Related Malaise Related Blood chemistry abnormal Related Clinic - General Classification Potentially Unrelated Atherosclerosis of coronary artery without angina pectoris Unrelated Gastroesophageal reflux disease without esophagitis Unrelated Benign essential hypertension Unrelated Room & Board - Semi-private Two Bed (Medical or General) - General classification Potentially Unrelated Medical/Surgical Supplies and Devices - Other Implants Potentially Unrelated Occupational Therapy - General classification Unrelated Pleural effusion Potentially Unrelated Pure hypercholesterolemia Unrelated Low back pain Unrelated Other Diagnostic Services - Peripheral Vascular Lab Potentially Unrelated Long-term current use of anticoagulant Potentially Unrelated Laboratory Pathological - General Classification Potentially Related Disease of liver Related Thrombocytopenic disorder Potentially Related Occupational Therapy - Evaluation or Re-evaluation Unrelated Cough Potentially Unrelated Operating Room Services - Minor Surgery Potentially Unrelated Medical/Surgical Supplies and Devices - Non Sterile Potentially Unrelated Nausea and vomiting Related Hypokalemia Potentially Related Pharmacy - Drugs incident to Radiology Potentially Related Gastro Intestinal Services - General Classification Potentially Unrelated Heart failure Unrelated Hypo-osmolality and or hyponatremia Potentially Related Cirrhosis - non-alcoholic Related Asthenia Related Hypothyroidism Unrelated Iron deficiency anemia Potentially Related Echocardiology Potentially Unrelated Chronic kidney disease stage 3 Unrelated Pharmacy - Other Pharmacy Potentially Related Chronic kidney disease Unrelated Treatment/Observation Room - Treatment Room Potentially Unrelated
This gave me a lot of new ideas, just listing a few
- Create a web of knowledge of concept relationship that is based on expert opinion of the LLM and the concept prevalence + patient context/co-occurrence data?
- Can we use this as a service to help phenotype development and evaluation - including case adjudication
(ok i am tired, i wish i could ask LLM to write this)
