Those ‘optional’ filters wasn’t part of the original algorithm that I wrote, and someone added it to account for bad data in the ETL (which I am an advocate of people fixing their data, not having all other analyses work around it ).
You are correct that applying those filters will remove records with null values. those filters should have been
AND (d.days_supply is null or days_supply >= 0). But, as I said, those were just suggestion filters, and not part of the ‘standard’ era building logic.
d.drug_concept_id != 0 filter is redundant, because the drug era logic finds the ‘Ingredient class’ of the drug exposure to roll up to, and conceptID = 0 does not roll up to an ingredient, so records with concept_id = 0 won’t be selected.
I’m hesitant to say what records ‘SHOULD NOT’ be used, rather the algorithm just specifies which records will be used:
- any record whose drug_concept_id rolls up to an Ingredient
That’s it. If they want to introduce a new rollup (they mentioned a CVX which is a sort-of ingredient level grouping?) then they just add that to the query to associate drug_exposure records to the correct concept to group with. I think we should be careful with this because the drug_era currently has a nice-and-comfortable understanding about rolling up to a specific level of the vocabulary hierarchy (ingredient) and I’m not sure what un-intended consequences we will get if we throw arbitrary rollups into the era table.