OHDSI Home | Forums | Wiki | Github

Evaluating the performance of a causal platform

Researchers
#1

Hello,

We’re developing a general-purpose causal inference platform to estimate drug effects on a large scale, primarily using a new user cohort study design. As a first step, we’d like to assess its performance using an established ground truth set, and consider using the OMOP drug safety reference set and replicating the results in Ryan at al’s paper (link). We’d appreciate any help and thoughts re the following:

  • For a given target drug, the control cohort (or comparator group) is defined using an active comparator drug (or set of such drugs) which “shares the same indication as the target drug but falls within a different therapeutic class”. What vocabularies were used to determine drugs’ indication and therapeutic class?
  • AFAIK indication and therapeutic class are characteristics of drugs, but the reference set’s concepts are ingredients. So, what drugs are considered for a given active ingredient, all or some subset (e.g., having an ingredient as the single active one)?
  • Also, the mapping of drugs, and even more so ingredients, to indication/therapeutic class is a one-to-many mapping; does “same indication” requires that the target and comparator drugs have identical sets of indications or just that they share an indication? Similarly, does “different therapeutic class” mean disjoint sets or non-identical?
  • Finally, is the specific list of the drugs used in each comparison available and can be shared?

In addition, we’d be happy to learn of any additional reference sets that can be used, or studies we can try to replicate, to evaluate our platform performance.

Thanks!

#2

None. Was done manually. I know what you want: You want to automate this selection. This was discussed a few times before, like here.

The problem is it doesn’t work very well. Because these indications are not at the same level that usual
diagnostic codes have. I give you an example. There is an antibiotic amikacin. One of the indications is “Pediatric fever without a source”. There is no such a SNOMED concept, because the Condition is “fever”, and the fact that is pediatric is part of the demographic information (year of birth less than 18 years away from current year). So, it gets mapped to “fever”. There are tons of drugs to treat fever, including aspirin, diphenhydramine, caffeine. Hardly comparators for an i.v. antibiotic.

So, nice idea, but won’t work.

Exactly. Ingredients can have very different indications depending in which combination they are.

Another good idea, but this stuff is not clean enough for any such heuristics to work.

I’d say go for the Legend project.

#3

We’ve done a lot of work establishing the OHDSI Methods Benchmark. Currently the only description is these two posters, presented at the 2017 OHDSI US Symposium:

We are working on a full paper, but it taking some time to get it accepted. The Methods Benchmark is implemented in the MethodEvaluation package, as described in this vignette.

With respect to LEGEND, I recommend reading our recent paper, although much more information on the methods can be found in this video recording of our presentation at the 2018 OHDSI US Symposium.

#4

Thanks @Christian_Reich and @schuemie for the links and helpful insights. Re the 2013 drug safety work, can you share the drug(s) used as comparator(s) for each tested ingredient?

Thanks.

#5

I think you’re looking for the omopReferenceSet data set in MethodEvaluation package:

data("omopReferenceSet")
head(omopReferenceSet)

  exposureId exposureName outcomeId                outcomeName groundTruth indicationId                          indicationName comparatorId   comparatorName                                           comparatorType
1    1309204    Adenosine 500000301 OMOP Acute Liver Failure 1           0     21001538 Paroxysmal Supraventricular Tachycardia      1135766    Phenylephrine     Target Primary Indication->Comparator Any Indication
2    1103552  almotriptan 500000301 OMOP Acute Liver Failure 1           0     21001116                                Migraine      1125315    Acetaminophen     Target Primary Indication->Comparator Any Indication
3     919204     Amylases 500000301 OMOP Acute Liver Failure 1           0     21002227                Pancreatic Insufficiency      1707164    Metronidazole         Target Any Indication->Comparator Any Indication
4     917006   Benzocaine 500000301 OMOP Acute Liver Failure 1           0     21001752                             Sore Throat      1192710 Chlorpheniramine Target Primary Indication->Comparator Primary Indication
5    1136601  benzonatate 500000301 OMOP Acute Liver Failure 1           0     21003205                                   Cough      1163944      Guaifenesin Target Primary Indication->Comparator Primary Indication
6   19008009  Cosyntropin 500000301 OMOP Acute Liver Failure 1           0     21000636      Acute Adrenocortical Insufficiency      1541079    Corticotropin         Target Any Indication->Comparator Any Indication
#6

Thanks, @schuemie.

t