OHDSI Home | Forums | Wiki | Github

Repo Request: CDM Atlas Cutover Package

Hello,

One of the takeaways from the OHDSI F2F was that some sites are refreshing multiple CDMs into new database versions on a frequent basis. When in this situation, it can become tricky or cumbersome to re-establish all of the OHDSI results tables/table conventions correctly in order to run Atlas. At Janssen, we use a “Cutover” R package that can handle this process, making it easier and more standardized to prepare our new CDMs to work with Atlas.

I’d like to get a GitHub repo (perhaps named “CdmAtlasCutover”) specifically for this process so that we can collaborate and improve upon this a bit more across sites.

Thanks,
Ajit

Very interesting. What does the Jansen cutover package do precisely?

Nothing crazy, but it is nice to have when cutting over new CDMs for Atlas use at scale:

  1. Creates OHDSI cohort results tables (cohort, cohort_inclusion, cohort_inclusion_result, cohort_summary_stats, cohort_inclusion_stats)
  2. Creates Heracles tables (heracles_results, heracles_results_dist, heracles_heel_results, feas_study_result, feas_study_inclusion_stats, feas_study_index_stats)
  3. Creates IR tables (ir_strata, ir_analysis_result, ir_analysis_strata_stats, ir_analysis_dist)
  4. Clears Atlas cohort results to ensure it’s clear that the cohorts need to be re-run, and not show there’s 0 patients: (cohort_generation_info, feas_study_generation_info, ir_execution, cca_execution).

We also set some permissions, but I’m thinking that may be best to leave out of the OHDSI package as that’s more site-specific.

There is a very old, long-standing issue about having flyway handle this migration:
https://github.com/OHDSI/WebAPI/issues/77 and https://github.com/OHDSI/WebAPI/issues/115

But there’s been zero movement on this. So in absence of an integrated solution, having an R package to create those tables in the result schema makes sense. We’d have to provide a dialect-specific translation in cases where sqlRender can’t handle the DDL.

The nice thing about flyway migrations is that you can preserve existing state between database changes, but this might not be as necessary for the results schema since you can always re-execute an analysis to generate the prior results. But for things maintained in WebAPI, the flyway migration maintaining state between migrations is crucial.

-Chris

Thanks. Sound like something that may be useful to add to Achilles or to Atlas. Thoughts?

It could be a part of Achilles, as the recommended convention is to always run Achilles prior to releasing a CDM for downstream usage. But I wonder about the scenario which we’re starting to face, which is where results tables are all being created in a separate schema. We tend to run Achilles in a development environment, so to have the Achilles run its various data quality steps and also create results tables / clear cohort results, it would need to have access to both the dev CDM schema and the production results schema. I’m not sure I’d be comfortable with that.

As a set of R functions, I would suggest we keep them in a separate package like the one I’m proposing. The long-term view, however, is probably what Chris has suggested with Flyway scripts, so that the WebAPI stack itself can drive all this after sources are refreshed.

Go for it. We can make the decision to fold into Atlas or port to Flyway later.

Great! As I don’t have permissions to create a repo in OHDSI, would you be able to create one for me (“CdmAtlasCutover”) and grant me the repo admin rights?

Neither do I. :slight_smile: @chrisknoll

1 Like

Nope, me neither!

Over to you, @Patrick_Ryan

Ok - Repo created and @Ajit_Londhe as been given admin rights.

as the recommended convention is to always run Achilles prior to releasing a CDM for downstream usage.

This is an interesting convention. In fact, I proposed that Achilles_results tables should become “derived tables” just like a condition_era table.

If this convention is truly followed at all sites (which makes a lot of sense) I like what @t_abdul_basser suggested about integration into Achilles. (also given the story of Atlas and integration of several tools)

t