OHDSI Home | Forums | Wiki | Github

ACHILLES "100% Change in Monthly Counts"

Developers
#1

Is anyone doing anything when they get these ACHILLES warnings? Really until recently we have been ignoring these warnings but wanted to know if someone has gotten value out of them or found something interesting because of them.

Here are some examples:

WARNING: 402-Number of persons by condition occurrence start month, by condition_concept_id; 3423 concepts have a 100% change in monthly count of events
WARNING: 420-Number of condition occurrence records by condition occurrence start month; theres a 100% change in monthly count of events

Tagging @Vojtech_Huser, @clairblacketer, @Ajit_Londhe who might also be interested.

#2

Still interested in if anyone has any use for these 100% change. If not, could I propose we get rid of them?

#3

This is for concepts that died or got remapped? And therefore 100% of them changed?

#4

It doesn’t indicate the cause, just the pattern in the data that there was a big change in concept counts. But it does spur to ask the question that you asked, @Christian_Reich: was it a mapping problem or a source data problem?

#5

Our team is working on trend analysis. (for research purpose and for Data Quality purpose). We are piloting use of methods that go beyond SQL statement.

Some ideas:

  • Perhaps we can change them into a notification priority and keep them. I am surprised that a group that developed them is now proposing to delete them.

  • Now when an updated API is out, we can perhaps improve the drill down feature of Achilles Heel. (inside DataSouces “tab” in Atlas). (if we find a web developer excited about this)

#6

@Vojtech_Huser - hahaha well @Patrick_Ryan suggested them initially and I can see what he was thinking but I have not spent much time thinking on how to take action on these in practice . . . . and I was hoping someone else could show me what they did or how it ended up helping them. If they don’t provide value I would suggest getting rid of them since it is too easy for them to pop up and I be default just ignore them.

#7

I am not suggesting these are the ideal metrics but they are useful for
one thing. When EHR data is compiled from multiple feeds (e.g., by
channeling HL7 router data or by a nightly dump from some system) thees
feeds sometimes go down. It can sometimes take weeks before the issue is
addressed. The implication for the EHR dataset is that certain values
dependent on the feed will drop significantly when the feed ends and
then jump up again when it returns. The measure “100% Change in Monthly
Counts” could identify that and call attention to time periods when
certain data is not really useful for analysis. Another example is that
some data might come from a survey or psychometric instrument that is
periodically filled out for patients. Sometimes changes are made to
important values. For example, the CMS Minimum Dataset used the nursing
home used a particular code to indicate exposure to certain drugs within
7 days prior to data collection by a nurse. In our dataset we saw a big
drop after two years across all sites. This was because of a code change
that CMS made in the survey but that was not noticed by the folks who
extracted the data from the nursing home database. This “100% Change in
Monthly Counts” picked that up and prompted us to look into why.

1 Like
t