OHDSI Home | Forums | Wiki | Github

Phenotype Phebruary Day 1 – Type 2 diabetes mellitus

This is great, thank you @jswerdel ! For those who haven’t yet played with PheValuator, here’s Joel’s initial paper on it in JBI, and here’s the codebase. He’s also presented several enhancements and benchmark studies at the last couple OHDSI Symposium, and that work is worth checking out.

It is tremendously valuable to get estimates of sensitivity, specificity and positive predictive value. for a given phenotype algorithm. Often with traditional chart review, we only get an estimate of positive predictive value, and if that’s all you’ve got, you can’t actually do any correction for measurement error.

For T2DM, it’s really interesting to see that the PPV is high (>84% for all three algorithms in both databases), but the sensitivity is more modest. This suggests that our algorithms, all of which require a diagnosis code, are missing over half the cases of T2DM. These databases don’t provide complete lab measurements, but you could imagine increasing sensitivity by creating a definition which allows persons with a diabetes drug (even without diagnosis) or persons meeting the ADA diabetes diagnostic criteria based on glucose or HbA1c values. Of course, any approach to increasing sensitivity needs to come with a proper assessment of the impact on specificity (and PPV).

@jswerdel , could you discuss briefly how you parameterized PheValuator to obtain these results?

@Patrick_Ryan this rule of requiring 365 days prior observation makes intuitive sense. From my clinical experience, adults who feel apparently healthy don’t find a reason to seek care. Type 2 Diabetes Mellitus is something that is indolent - i.e. the person feels “apparently” healthy - and probably has the disease without knowing it for a long time. But on the contrary, once a person receives the “label” of Type 2 Diabetes Mellitus - they are more likely to seek follow-up care.

This 365 days prior observation time - trys to tease out the people who are being observed in the database for the first for initial diagnostic care for Type 2 Diabetes Mellitus vs those who are receiving (follow-up management care for Type 2 Diabetes Mellitus.

But why 365 days - why not 1000days or 100 days? Is this an OHDSI best practice or is this a research opportunity?

I think it would be wonderful if Cohort Diagnostics could tell us the characteristics of the people that are in C3 but not in C2/C1. What are the attributes of people we are loosing? Are the population level characteristics of the people we are loosing similar or different from C3?

i.e. are the people we are removing less likely to represent the type of clinical profile described by the clinical description from the American Diabetic Association. If yes, thats perfect!

1 Like

@Patrick_Ryan @hripcsa I think the clinical description should be clarified to make progress on this topic. One definition of incident is the first time the Doctor or the patient learnt that the person has the phenotype. Another definition is the first time the person biologically had the phenotype.

If we are refering to the first time it was learnt that the person has the phenotype: then the 365 days rule would be appropriate.

If we are refering to the first time the person biologically had the phenotype - then the clinical description could clarify that if the person also has indicators of disease chronicity such as diabetic foot, paresthesia, ulcers etc - that is not new onset diabetes. In this case, we will need to add additional inclusion rules to the cohort definition – so as exclude people with such indicator of chronic disease between day 0 and upto some future days as allowed by clinical description (e.g. if it not expected to have diabetic foot within 1 year of biological disease onset)

The clinical description is not asking for biological incidence - as written

@Gowtham_Rao , important point. I probably should have stated it more generically as:

“we add an inclusion criteria that requires some period of prior observation, with the intent to give confidence that the event is new because it hadn’t been previously observed for that prior observation duration”

365 days is purely a convenient heuristic commonly used, but it has no real empirical basis and likely is highly inappropriate in many circumstances: it is probably too short if it would be reasonable to expect a person wouldn’t go back to seek follow-up care every year because it can be managed effectively by the patient (mild arthritis and osteoporosis come to mind), and it is probably too long if it would be reasonable to expect very regular care (like end-stage renal disease, where you’d expect to see monthly dialysis). I’ll also note that the COVID pandemic totally screwed with lots of regular preventative service/well visits, so the gaps between normal care could be longer in 2021-2022, that what we may have seen previously. (Anyone in the community have a database of dental visits to plot this out? :slight_smile: )

The decision here effectively amounts to a bias / variance tradeoff. A shorter ‘prior observation’ value will increase the chance that you are pulling in ‘prevalent’ cases into your ‘incident’ case definition, which means you’ll have greater index date misspecification. But, you’ll also have a larger sample size which will increase your statistical power for whatever question you are trying to answer (which is the argument I most often here, when people like to diddle around with this number from 365d to 180d). A longer prior observation window will increase your confidence that cases are truly incident, but then you may actually be excluding some ‘true incident’ cases simply because they don’t enough historical data.

I do think some empirical investigation could be do into looking at the impact of this design choice. Off the top of my head, I think you’d probably start by creating some evaluation set of persons who did have some extended observation period time (like, for argument sake, 10 years). Then you’d apply phenotypes with different prior observation period lengths within that subset, and you’d be able to compare the resulting patient sets. I don’t have any intuition for how big an issue this actually is, but my initial gut is that 365d could be a bit low for what we may want when services involve annual check-ups and patients may miss or delay a follow-up visit.

I agree - this is an opportunity to test the impact. I will try to visit this after phenotype phebruary thru another OHDSI Workgroup

An old Jim Lewis paper worthy of review regarding identification incident events worthy of review/replication in today’s data. The relationship between time since registration and measured incidence rates in the General Practice Research Database - PubMed

Another consideration is that the requirement of having X many days of observation before the diagnosis forces us to drop persons with shorter observation period before their diagnosis. This can be a problem in US data, where follow-up is truncated when persons change jobs or health plans. Palmsten et al showed this clearly in the context of drug safety in pregnancy using Medicaid (the observed effect is possibly more extreme than in other data sources); see figure 4 in Harnessing the Medicaid Analytic eXtract (MAX) to Evaluate Medications in Pregnancy: Design Considerations

As I was preparing to discuss the parameters for running PheValuator for this analysis, I was concerned about the low sensitivities and wondered if this could be explained. I experimented with different parameters for the evaluation cohort (to be explained below) and got higher sensitivities:


while the PPV’s remained about the same. With that in mind, let me briefly review some of the parameters (for the full explanation, please see the vignette). The process has been changed significantly since V1. In the latest version. we use visit level analyses for estimating the performance characteristics as compared to using all the data in the subject’s record in V1. Details of the xSpec and xSens cohorts are in the vignette and are a bit too lengthy to discuss here. The changes I made were in the evaluation cohort, the cohort with a large random set of subjects either with the condition of interest or without it. In the original analysis I created an evaluation cohort where a random visit in the subject’s record was selected for analysis, including for those with T2DM. However, the algorithms we wanted to test were for the earliest recorded diagnosis for T2DM. I changed the evaluation cohort to only include the earliest visit for those with T2DM. Using this approach increased the sensitivity as shown. Subjects were now being matched better on a visit by visit basis. We had observed lower sensitivities in our PheValuator Benchmark comparisons and this may help to explain that finding.
One other interesting finding, when I changed the first algorithm to not include the requirement for a 365 day lookback (the fourth algorithm in the list), I found a higher PPV compared the the original. Subjects in the prevalent algorithm, on average, may have a higher probability of being a case compared to newly diagnosed subjects. These subjects are more likely to be well into their treatment so the diagnostic predictive model used to evaluate the subjects has more evidence to support the diagnosis and estimates a higher probability of the condition.

@Gowtham_Rao Hi, sorry for joining the discussion late. Not sure whether it has already passed the registration deadline. I filled the form 2 days ago but didn’t receive any email to guide me on how to create an account to access the altas-phenotype.ohdsi.org. Please let me know if something I am missing here.

@Helen , you are definitely not too late (though I do appreciate that you’ve used a sloth as your icon for this post :slight_smile: ). Phenotype Phebruary isn’t stopping, we’re just using it to get the conversation started. You are register for access to atlas-phenotype by filling in this form: OHDSI Atlas Phenotype Library Registration . And also, I encourage you to join the Phenotype Development/Evaluation WG, which is run from within the OHDSI MSTeams environment, and you can sign up for this and any other WG you’d like here.

Thanks, Patrick. I already filled out the form last week but haven’t received an email about creating an account yet to access the resource. Do you have anyone to suggest to reach out to? Thanks ahead.

@Helen There are two different forms. One is to sign up for the Work Group, the other form (see link below) is for atlas-phenotype.ohdsi.org registration.

There is no record that you completed the form for atlas-phenotype.ohdsi.org registration.

Please use the link below to submit the atlas-phenotype.ohdsi.org registration form and your atlas account info will be mailed to you.

When I opened the link, it shows that I already filled the survey. Can you check it again? The email I used is hetinghelen@gmail.com, and I am from JHU SOM.

@Helen your Atlas Phenotype account has now been created and sent to you via email.

First of all, thank you for this impressive work! That was a joy to follow phenotype phebruary from the very first day.

However, I think that there is an issue we have in this phenotype resulting from combination of Atlas behaviour and Vocabulary hierarchy. Using phenotype 1 with conditions only, I noticed that in Atlas included source codes, there are a few concepts with type 1 diabetes.

There are none of type 1 diabetes in included standard concepts, which suggests erroneous mapping. Lets take one code as example and see:

E10.10 Type 1 diabetes mellitus with ketoacidosis without coma is mapped to 2 conditions: 201254 Type 1 diabetes mellitus and 4009303 Diabetic ketoacidosis without coma, which is neither connected to Type 1, nor to type 2 diabetes, which is true, because any type of diabetes may result in diabetic ketoacidosis without coma. What is not true is that patients are included in Type 2 diabetes cohort despite explicit exlusion of Type 1 diabetes from the concept set. These codes were included by 442793 Complication due to diabetes mellitus and not excluded by 201254 Type 1 diabetes mellitus, which I think is a bit tricky of Atlas.

The pretty close issue is discussed right now in another topic.

We excluded Type 1 concepts by source code and the cohort was limited to only Type 2 DM patients.

So is it the expected behavior of Atlas and it just should be kept in mind or should be changed?
Am I missing something? Thoughts?

@Gowtham_Rao @Patrick_Ryan @Christian_Reich @Chris_Knoll @aostropolets @Alexdavv

Without knowing what the origin concept set expression is, I can’t comment on which concepts may be improperly mapped.

What I can say is that Atlas isn’t applying any special logic on top off the CDM vocabulary tables. When yous ay:

I’m not sure what trickery you’re referencing but when looking at the included source codes, you’re simply looking at the source codes that map over (via source-to-concept-map) to the standard concepts that are included from the concept set expression.

To put it another way (on how we get to included source concepts) we:

  1. Resolve the concept set expression into included conepts (uses CONCEPT_ANCESTOR)
  2. From the result of 1, find concepts via source_to_concept_map.

If the concept set expression wanted to include complications and excluded T1 concepts, then it makes sense that you got the complication included that happens to reference a T1DM cause…because the concept set expression is puling in any concept that’s a complication but the complication itself is not a T1DM diagnosis…

This is just some of the interesting side-effects you get from using the CDM vocabulary, and this is specifically why we give you the ‘included concepts’ and ‘included source code’ views in Atlas.

The likely conclusion here is that the vocabulary mapping is correct, and the concept set expression has to be more specific about which complications you want to find (such as only complications related to T2DM). But this is really a wild guess as I don’t know the specific code list and clinical concept you’re trying to model.

Thank you for the answer.

Type 2 diabetes, definition 1 has a concept set of conditions I am talking about. And I believe that concepts are mapped properly.

I also understand the mechanics behind it, what I found tricky is the side-effect. Anyway, the source codes referencing type 1 diabetes should be excluded from the mentioned definition.

If this effect is known, I just wanted to make sure that I am the only person considering it tricky :slight_smile:

I played around with this, and I think that it’s a common trap where you think that if you include all disorders related to DM, but then exclude the ones related to T1, you would just be left with only those related to T2. However, as you saw, the 4009303 is something common to both T1 and T2 so you would get both source codes that indicate T1 and T2. So, IMO, using the ‘Complcation due to DM’ (ConceptID: 442793) is a little over broad because some of the descendants could be from Type 1 codes, and there’s nothign in the hierarchy that indicates that the Type 1 code falls under ‘complication due to Type 1 DM’.

I was able to remove the code by removing the general ‘complication due to DM’ and replaced it with ‘Complications due to Type 2 DM’, but I’m not sure what other side effects that may have, so I’m not suggesting anything related to the official T2DM phenotype. But this is a good excercise to show how using concept ancestor in the vocabulary may not always give you what you thought, usually because there’s a broader definition in the hierarchy that doesn’t have the type-specific children beneath it.

There is a concept for ketoacidosis due to type 1 diabetes (with a child that indicates ‘coma’) but there isn’t one that is specifically ‘without coma’…I don’t know if it makes more sense to map the E10.10 to 439770, but I’m pretty sure we got these mappings from SNOMED and so there was probably a reason to map to 4009303 vs 439770. But, that’s only half the puzzle because it doesn’t appear that 439770: Ketoacidosis due to type 1 DM roll up to ‘Disorder due to type 1 dm’, so it wouldn’t fully help us for purposes of how we are trying to define T2DM in this phenotype.

My 2 cents although with inflation it might not be worth what it use to be. You have to have the disease before you can have complications. How many people are entering a T1DM or T2DM cohort based solely on a complication code as the initial presenting diagnostis? Now yes it is very true that patient’s first experience with a condition may be as a result of a complication of said underlying disease which as of yet has not been properly recorded. I send my insurance company a quarterly HOMA-IR value so we will know precisely the first quarter upon which my beta cells starting deteriorating.

t