OHDSI Home | Forums | Wiki | Github

Quarterly CDM Refresh in PostgreSQL - Risk of Losing ATLAS Definitions?

Developers
,
#1

Hello everyone,
We manage an OHDSI ATLAS instance with 6 CDMs databases from 6 different hospitals, the CDMs are on PostgreSQL. we are refreshing the data on a quarterly base, as a full data load.
The full process of uploading new data and executing the Achilles reports takes 8-10 hours per hospital, which means 60 hours of overall ATLAS downtime.
To avoid downtime during the refresh, we are considering the following process:

  1. Load the new data into a new CDM.
  2. Run Achilles reports on the new CDM to generate a new Results schema.
  3. After the first two steps are complete, switch the mapping in the ATLAS configuration the new schemas.

Our questions are:

  • Could this process cause us to lose Cohort definitions or Concept set definitions?
  • Does anyone have experience performing a data refresh in this way, especially in a PostgreSQL environment?

I would appreciate your thoughts and recommendations.

Thank you!
Guy

#2

Hello @guy_livne,

Could this process cause us to lose Cohort definitions or Concept set definitions?

The definitions of the concept sets and cohorts are stored separately in internal WebAPI database, so they won’t be lost. The cohort generation results, however (i.e. the number of patients), are stored in the Results schema of your PostgreSQL instance and will be overwritten. (But it sort of makes sense because the OMOP data they were based on has changed.)

After the first two steps are complete, switch the mapping in the ATLAS configuration the new schemas.

Or you could simply add the new version in ATLAS alongside the old one:

  • there’s no ATLAS downtime;
  • users can migrate their studies and compare the results in old vs new data;
  • having separate Results schemas for separate versions means that no ATLAS results are overwritten;
  • old version can be removed immediately or phased out after some time;
  • even if some critical issues are found in the new version, you only have to delete a data source entry in ATLAS configuration to revert to the previous one.
#3

Great, thanks @rookie_crewkie

t