| OHDSI Home | Forums | Wiki | Github |
Today and for this week, we want to start a discussion about Peer Review. In particular, we want to start answering the following questions:
On top of checking the accuracy of the clinical exposĂ©, arenât the criteria sensitivity, sensibility, timelines and correct exit? Isnât that what we need from a phenotype?
Thanks @Azza_Shoaibi for kicking off this Phenotype Phebruary 2023 week 1 discussion topic. And thanks @Christian_Reich for chiming in with your perspective. I really hope weâll hear from others on this important topic.
I really appreciate the careful thought and consideration that @Evan_Minty and @Andrea_Noel put into their peer reviews on pancreatitis and anaphylaxis, respectively, as well as the great discussion that took place on the interactive peer review of appendicitis.
That said, Iâm still myself undecided about what (if any) role a peer review should play in the phenotype development and evaluation process.
My current thoughts (which Iâll try to share here provocatively in the hopes to stimulate some debates with others in the community who may have difference of opinions):
A phenotype algorithm (aka cohort definition) is a specification for how we identify a cohort, which is a set of persons who satisfy one or more inclusion criteria for a duration of time. For ease of simplicity in this discussion, Iâll focus on the task of phenotyping a disease.
For a given disease (which needs to be precisely defined in a clinical description), the task of the phenotype algorithm is simply to use the data available to identify which persons have the disease and when.
We can try to measure the extent to which a phenotype algorithm accomplishes this temporal classification task in a given database. There are three primary types of measurement error that we want to be the lookout for:
Given a database estimate of sensitivity/specificity/positive predictive value and a quantification of index date misspecification (e.g. what % of persons have incorrect date?; what is the expected duration between real and assigned date?), one should be able to evaluate if the phenotype algorithm is adequate for use in that database by determining that calibration for the measurement error on the evidence statistic of interest (such as: incidence rate in a characterization study; relative risk in a population-level effect estimation analysis; AUC and calibration in a prediction study) is sufficiently small that it will not substantially impact the interpretation of the evidence. More ideally, this would be an objective diagnostic, with pre-specified decision thresholds, such that phenotype algorithm adequacy could be determined explicitly by the measurement error estimates. For example, a decision threshold could be something like: accept phenotype algorithm if ABS ( LN (positive predictive value / sensitivity) ) < 0.69; else reject phenotype algorithm. This decision threshold ensures that calibration of the incidence rate of the phenotype as an outcome in a characterization study doesnât result in more than a 2x change in the observed incidence in either direction.
If we could agree to objective diagnostics and pre-specified decision thresholds to determine phenotype algorithm adequacy based on the measurement error estimates for a given database (and its still a big âifâ that requires more methodological research like what @jweave17 is doing with @Daniel_Prieto at Oxford), then what is the role of peer review?
In my opinion, the job of the peer reviewer is NOT to question the choices within the algorithm, to review the codelist, to speculate on ways the algorithm could be tweaked, or to project whether the algorithm should be used by other researchers in other databasesâŠall of these tasks are simply pitting one personâs subjective opinions against another, with little-to-no empirical basis to reconcile differences. Instead, I believe the role of the peer reviewer COULD BE two-fold: 1) to review the clinical description and ensure it is not ambiguous about the intended target of the phenotype algorithm, and 2) to verify that the phenotype algorithm measurement error estimates are appropriate measures aligned to the clinical description. Akin to peer review of a journal, the options of a peer reviewer would be to either a) accept the phenotype algorithm into a library as a valid algorithm worthy of consideration for re-use by future researchers, b) reject the phenotype algorithm because the measurement error estimates are not valid, or c) request a ârevise-and-resubmitâ to improve the clinical description or the accompanying evidence about measurement error.
Regardless of whether a peer review takes place, it seems clear to me that the role of the researcher who is conducting a new study and needs to decide if she will re-use an existing phenotype algorithm is to: 1- determine if the clinical description of the algorithm aligns with her research question, and 2- determine if the measurement error in her data network is sufficiently small to move forward. For the researcher to accomplish #2, she either needs to 1- re-use the same data that the measurement error estimates were originally generated on, 2- produce new measurement error estimates on her own data, or 3- make a generalization assumption that the error estimates observed from the original developer are likely to be consistent with what would be expected to be observed in her new data (without empirically evaluating that assumption). Given this responsibility of the researcher, itâs unclear to me how having a peer review complete ahead of time helps, aside from potentially giving some superficial confidence in considering the phenotype algorithm in the first place.
For phenotypes that truly desire to have first occurrence of a condition within an individualâs lifetime our resources will need to enhance the ability to link records to advance the ability to computationally characterize a personâs person-time. Some individuals in a database have eight plus years of follow-up time others have the request single trip around the sun worth of data collection.
So my clinical review of a clinical description for {insert new onset chronic disease here} will likely focus on the ambiguity around new to the data or truly new to the patient based on available database follow-up time. Perhaps we should also consider crafting limitations sections to our phenotypes. Similar to my peer review of journal articles when I get to the limitations section and read âwe studied 30d hospital readmission but we only know if they came back to our hospital.â We could have some authored idea around guardrails limiting future researchers over extending the phenotype capabilities. That said, I totally agree that it is the role of the future researcher to determine if the existing phenotype is fit-for-purpose for the new research question or different database.
So that said, do we need the peer review process or the pre-print model which this Phorum (itâll catch on! we pharmacists love spelling things with Ph, itâs acidicâŠ) so provides our community with the open source dialogue on our algorithms.
In the presence of objective criteria, the role of peer reviewer would be limited or not needed. Today we do not have objective criteria or such methods are not feasible (eg chart reviews).
The challenge we are trying to help solve is trust our phenotype (and so trust in our evidence) - what would make the work of one scientist trustable by others. In the absence of objective measures of such trust we need subjective judgments. By soliciting another peer scientist to provide their independent perspective - we hope to discover measurement errors that we would not have otherwise seen (or did not care to see).
At the minimum, a peer reviewer may look for
In our field, no cohort definition is perfect - and there will always be measurement errors. If we go ahead and perform research without being aware of such errors, our work may suffer from unknown biases - ie less trust worthy.
What I think the peer scientist is doing is serving as an independent check to see if the original contributing scientist knows the (potential) measurements errors.
That is how we operationalize the phenotype. The criteria are the the means for that. The purpose of a cohort is to create a set of persons that satisfy the clinical description.
Now, what is a clinical description? It is not a textbook narrative of the ideal patient, like what we are producing a lot here. Instead, it is a biological state of health of the organism of the patient. How do we know that? We take the word of the clinical in the data, pressure test it, and recruit additional data to help with the misclassifications (see below).
Exactly right. But what we do have is having clinical knowledge to make a good guess about misclassifications and the criteria we invent to correct for that.
Which is why we need to operationalize, or engineer this process.
| Step | Job of the peer to pressure test |
|---|---|
| 1. Define the index criterion | Is the conceptset checked for sensitivity (forgot something) or specificity (too broad) errors. |
| 2. Decide if sensitivity is a problem | Are all reasons considered: (i) disease not reimbursable, (ii) disease common and not severe, (iii) disease hard to diagnose, (iv) disease often confused with differential dx, (v) codes lousy. |
| 3. Decide if specificity is a problem | Are all reasons considered: (i) disease likely to be upcoded, (ii) diagnostic workup reimbursable, (iii) disease hard to diagnose, (iii) disease often confused with differential dx, (iv) codes lousy. |
| 4. Decide if onset timing is a problem | Are all reasons considered: (i) disease builds slowly, (ii) disease sensitivity low and therefore diagnosed late, (iii) disease not reimbursable. |
| 5. Decide if resolution timing is a problem | Are all reasons considered: (i) disease can progress differently, (ii) only way to tell is if patients stop showing up. |
| 5. Check out criteria fixing 2.-5. | Are cohort comparisons evaluated: with index criterion alone vs with criteria 2.-5. added, one at a time. |
We only have a chance if we apply the OHDSI principles of a systematic approach. If we do undefined âreviewsâ we add more pain to the current situation.
Thanks @Christian_Reich . The steps you lay out seem reasonable as part of the phenotype development process , and directionally quite aligned with what I tried to introduce on the Phenotype Phebruary kick off call (video here). Namely that we should try to get in habit of developing our phenotypes with evaluation in mind, specifically to have each step aimed at reducing a source of error:
I would assert that we can and should build better diagnostics for each of these 3 steps: 1- diagnostics that can help us identify sensitivity errors by finding concepts we havenât yet included which would increase our cohort size without substantially changing the composition of the characteristics of the cohort; 2- diagnostics that help us identify specificity errors by finding concepts which, if included as inclusion criteria based on a requirement of either their presence or absence, would decrease our sample size and would change the composition of the the characteristics of the cohort (under the premise that those now excluded are different people than those who remain); 3- diagnostics that help us understand the distribution of recurrence and the duration between successive entry events, so that one can determine how to differentiate follow-up care from new âincidentâ episodes of disease; and 3- diagnostics that observe the prevalence of symptoms, diagnostic procedures, and treatments in time windows relative to the assigned index date to determine what revisions to entry events may be warranted to reduce index date misspecification.
At the end of that development and evaluation cycle, if youâve iterated to increase sensitivity, then increase specificity, then decrease index date misspecification, on one or more databases, then you should be left with a final phenotype algorithm that warrants consideration from the community. In an preferred state, the sensitivity, specificity, positive predictive value, and index date misspecification would be explicitly quantified, but at a minimum, can be discussed in an evaluation report summarizing the algorithm.
For me, Iâm just unsettled on what the incremental value of the peer review process is. If itâs simply to verify that some process has been followed, then weâre just talking about a checklist to ensure the submission has the required elements (without subjective judgement about its contents). But if we imagine peer reviewers have to execute the development/evaluation steps themselves to see if they reach the same conclusion, then I think thatâs a unnecessary burden on the reviewer and quite unlikely to yield a successful replication. I think of it akin to peer review in publications: the reviewer is supposed to be assessing the scientific validity of the submission: that the methods are sufficiently described and appropriate to generate the results presented, and that interpretation of results is put in sufficient context with what is known and what can be inferred from the data. But they are generally not responsible for independently reproducing the results nor should they try to change the original aim of the study (though they can opine to the journal editors of whether they find the topic of relevance to the field). It seems to me, in our world of phenotyping, its ALWAYS appropriate to aim to develop a phenotype algorithm for a particular clinical target (so we donât need to question intent or relevancy), so what does that leave for the peer reviewer to do?
The OHDSI Phenotype Development and Evaluation Workgroup has made progress in providing guidance on clinical description (note this is still work in progress and in DRAFT stage), however the key idea .
In Phenotype Phebruary 2022 - I introduced the idea of clinical description as the most important step, and now we assert this is a required first step.
Reason for this assertion is: if we dont spend the time to understand the clinical target being phenotyped, then in the absence of the clinical understanding of a) what it is, b) what it is not, c) how does it start, d) how does it stop, e) how is it managed â we cannot recognize measurement errors in our
Today there is no debate on the importance of stating upfront the target clinical ideas description, but we are still struggling with focusing the content as needed by the phenotyper. Clinical description is like a case definition but at the population level and required scientific practice.
Clinical description is important for the peer reviewer. As seen in the interactive session - Appendicitis - peer review - because the clinical description stated upfront that the clinical idea of appendicitis included the spectrum of inflammation of appendix - the peer reviewers agreed that âCrohns disease of appendixâ was appendicitis. Clinical description justified design choices made by the phenotyper.
It is the clinical description that articulates the target clinical idea we are trying to model with our cohort definition and justifies design choices.
Soon, the Forum software is going to give me that infamous âThis subject seems to be really important to you. But consider other people also weighing inâ. Will happen soon. 
We hear you. I donât know either. Maybe it is a euphemism for ensuring that the process is indeed followed and all aspects are thought through? Because few people have the clinical knowledge of all conditions and procedures, AND also know how these facts end up in the data by the data capture system?
I find that example proves the opposite and is disconcerting. Because nominally that assertion is correct. But letâs say you want to study the effectiveness of one antibiotic (curative or preventative for the surgery) compared to another for some bad outcome (perforation or death) - you would never include Crohnâs, because thatâs just a completely different etiology. However, if you want to study, say, the correlation between inflammatory blood markers and perforation you might include Crohnâs very well. Which means: The research question determines the clinical description, and there is no phenotype without stating the question per se.
What does that mean for our work?
If we bring this two together I think we all agree that the sequence is:
Research question â define clinical idea â phenotype development and evaluation.
concretely e.g., if this is a clinical trial, we would say the experiment is âacute appendicitis among adult persons without crohns disease or chronic appendicitis randomized to either antibiotic A or antibiotic Bâ. Then you would create patient eligibility rules for the defined clinical idea âacute appendicitis among adult persons without crohns disease or chronic appendicitisâ such as should be adult, should have a new diagnosis of appendicitis, should not have chronic appendicitis or crohn disease history - which would go to sites that recruit patients. We would also follow the same sequence in observational data - replace patient eligibility rules to cohort definition logic.
If the above sequence is approximately correct, then the path to phenotype development and evaluation is
âdefine clinical idea â phenotype development and evaluationâ
i.e. there is thus no logical dependency between phenotype development and research question.
In the above example - you would specify in the clinical description to say - âpersons with appendicitis who have co-occurring crohns disease involving appendixâ are not eligible.
This clarification in the clinical description - would justify a rule in the cohort definition logic to remove persons with crohns disease involving appendix. It would also make it clear to the peer reviewer, why a certain inclusion rule was applied (i.e. exactly 0 persons with crohns disease of appendix).
Now the evaluation is geared towards the target of âacute appendicitis among adult persons without crohns disease or chronic appendicitisâ and not âappendicitisâ. i.e. finding evidence that there are persons in the data source who have âcrohns disease of the appendixâ but those persons are not in your cohort - would not mean sensitivity errors, while seeing evidence that there are persons in the cohort with âcrohns disease of appendixâ would indicate specificity error.
Compare to just âappendicitisâ - there the finding evidence that there are persons in the data source with âcrohns disease of the appendixâ and that those persons are not in your cohort - would be a negative hit to sensitivity.
âcrohns disease of the appendixâ is a good for appendicits, but bad for âcrohns disease of the appendixâ is guided by the clinical description iâŠe target clinical description which are different for âappendicitisâ and âacute appendicitis without crohns disease of appendixâ
@Christian_Reich , I think I have a different take on the appendicitis example.
My perspective: there are two completely valid clinical targets that could be unambigiously described in a clinical definition: 1- acute appendicitis without prior chronic inflammation, and 2- inflammation of appendix (acute or chronic). âCrohnâs disease of appendixâ may be included in #2 but would likely be excluded from #1 in the clinical description. But there is no judgement about which of these clinical targets is ârightâ or âwrongâ, both are legitimate and can be useful for different research questions.
For each of these clinical targets (with their associated descriptions), it should be possible to develop and evaluate a phenotype algorithm, and it would seem perfectly fine to me that an OHDSI phenotype library would house both target/algorithm pairs. I donât see any disconnect here.
Itâs the responsibility of the researcher to ensure they are picking the appropriate phenotype from the library that suits their research need, but we donât need to know that researchers question before we develop the phenotypes. In my opinion, this example seems more to point to the need to reduce ambiguity in the clinical description (and possibly ensure that our label for the phenotype is sufficiently descriptive so as to avoid confusion for future researchers).
One small addition. If we come up with thresholds for estimated sensitivity and specificity or PPV, I think that mainly addresses misspecification of the variance of the desired clinical estimate where the phenotype will be used. And I expect the thresholds need not be overly narrow (we can give some slack).
More important, and what I am beginning to work on, is differential measurement error that biases the desired clinical estimate. That requires knowing the clinical hypothesis. Right now at least, that is mostly a human review, be it phenotype developer or eventual phenotype user. I can imagine coming up with some heuristic number for estimated bias or sensitivity, but I would have a hard time really trusting it.
So we can consider it a post-phenotyping process, except presumably the better the sens. and spec. or PPV, the less opportunity for differential error.
Wow! This is a REALLY good discussion.
You all give me some good thoughts to improve/clarify the âpeer reviewâ step. As I was reading this I found myself agreeing with @Patrick_Ryan @Christian_Reich @Gowtham_Rao and @hripcsa all at the same time!!! Here are my few additional thoughts:
Sure. If in your clinical description you concluded the job of resolving all those potential choices, and made that clear in the text. But that means you canât run a Phenotype Phebruary and go âwho wants to do appendicitis?â There is no such a thing as âappendicitisâ.
Appendicitis is one example where your phenotype requires iterative workup, with potential splitting. The notorious AMI - ischemic heart disease - unstable angina - stable angina continuum is the same. Or the ischemic - hemorrhagic - ischemic not turning hemorrhagic stroke. Or the acute pancreatitis - cholecystitis - chronic pancreatitis. Or the anaphylaxis - anaphylaxis but not from an insect bite. And so on. These are only the examples I remember from the recent past. I guess every single Phebruary cohort has to be split into several ones - depending on what you want to study.
I think I am saying exactly the same thing. Yes, you can parse all those ambiguities and add them to the clinical description. Except you would multiply each library entry and make many out of them. Maybe thatâs what we should do: a hierarchical library.
The clinical description is 1000 times better than what we have now - wishy washy descriptions in the papers. But still they fall short of what is needed: The poor analyst wretch coming to the library has no idea what pitfalls appendicitis can have. And that he has to decide about the Crohnâs. And the chronic appendicitis. And maybe more.
Currently, the clinical description is a yada yada textbook summary of the disease, and how you manage it, and that the patient will be in the ER, and so on. Instead, it should say âAppendicitis, but no chronic conditionâ or âAcute appendicitis, including an acute eruption of a chronic conditionâ. Since we are doing this exercise, it would be a huge help to the community to be told what key choices they have to make.
Differential error between what? Different phenotype versions?
If for the same phenotype, the sensitivity and specificity differ by cohort (eg treatment group), then you can get biased estimates in whatever study you are doing. I guess basically, some aspect of the error would have to be a confounder, correlated with both treatment and outcome. We normally discount the possibility, but thatâs what would cause the most trouble.
@hripcsa are you referring to selection bias? i.e., If T has a sensitivity of 60 and PPV of 40, and C has a sensitivity of 99 and PPV of 99âŠ
Then there are personâs in T who donât have the disease/treatment but the cohort definition algorithm is classifying them to have it, ie they may have something else thatâs we donât want but we think they have something we want. ie measurement error.
In such scenarios, the measurement errors maybe so bad that we canât achieve exchangility state (like randomisation) with for example our propensity score methods. In the absence of such a state, there will be residual confounding causing a lot of trouble.
The question is what is the question. Atrial fibrillation as an adverse event may require a different phenotype from Atrial fibrillation requiring DOAC therapy. Thus necessitating additional clinical descriptions and additional reviews and diagnostic metrics.
We always need to start with the research question which may alter the clinical idea based on the research question and we may discover that our data is not fit-for-purpose for a given research question. If I donât believe you are diabetic until I see a HOMA-IR or youâre not depressed until I see a HAM-D then our data will not be able to define a clinical idea needed to address a highly specific research question.
On the topic of assessing differential misclassification how in rare exposures (T) and rare outcomes do we acheive the precision necessary to conclude that 60 and 99 are different. Now these are extreme differences but 70 and 80 are different but only if you have enough data to conclude they are not.
