OHDSI Home | Forums | Wiki | Github

Updating Atlas cohort defenitions not showing up

Upgrading from a very old version 2.50 to the newest 2.11

I have copied the source table exactly as it was in the old version and the connections work. However, none of the old concept sets are now showing up in Atlas.

I was hoping to be able to use all the ones that had been created for the years in which the old version was in production.

Is there any way to keep those old concept sets and cohorts from an old to new version?
Am I just missing something?

Any and all help is appreciated.

Best,
Alex

There should be no reason that the assets like cohort definitions and concept sets should be lost going from 2.5 to 2.11.

Could you query your webapi cohort_definition and concept_set tables? They should have data in them. The biggest change from 2.5 to 2.11 was a change in ‘autonumber’ fields, but without further information, I can’t tell. Can you provide details about what is in your cohort_definition and concept_set table and we’ll take it from there.

Hi Chris,

So I have a new webapi database that the updated instance is looking at. Looking at those tables they are empty in the new one and have all the data stored in the old one.

I had thought the cohort and concept data was stored in the ohdsi_results database but I guess that is incorrect. Thank you for the clarification.

This leads to another question is if it would be possible to use an older version of the webapi database with a new version of the webapi?

The old database was created alongside webapi version 2.5 and the new version of the webapi is 2.10

When I tried to point the new version of the webapi at the old database it spit out a number of SQL errors with the same error message:
ERROR: there is no unique constraint matching given keys for referenced table “source”

This along with the fact that there are a number of tables that don’t appear in the old version led me to believe that the two are not compatible.

If this is not the case please let me know.

Is there any good way to migrate the old data into the newly updated database?
ie. would it be as simple as taking all the data from the old concept_set and cohort_definition tables and any associated tables and putting them into the new database?

Thanks for your help!

Best,
Alex

Pointing the new version of the WebAPI application at the existing (old) webAPI database should have auto-migrated all tables (including creation and drop) to the newer schema in v2.10. If there was manual changes made to the WebAPI database, then the migration scripts will not know how to handle that and the migration scripts will fail.

The error message you provided isn’t clear enough without context (such as which migration script led to the error, or if it was some other error on startup).

If you want to try to migrate from one WebAPI instance to another, you could look into using ROhdsiWebAPI to query the old WebAPI instance for the existing assets and then post them to the new webAPI instance for loading.

You could also look at a more ‘raw’ database migration where you export the data from the old tables and then bulk-insert them into the new tables, but this is a very technically complicated maneuver where if you don’t import the data in the right order and use the right database sequences you could make a mistake that will cause the application not to work.

It’s also not clear which version of DBMS you are running on so I don’t know if there’s any database-specific information I could give you.

Worst case scenario is you keep the old WebAPI running in 2.5, and have the new application running on 2.10, and you can export-import from one application to the other. A way you could do this is if you create a cohort characterization, add a bunch of cohort definitions to it, and then export the cohort characterization, you can import the characterization design into the new instance and it should not only import the characterization design, but all the cohort definitions that were included. There’s nothing that would do the work for concept sets, however. Your best bet (if you want to keep the 2 instances running) would go with the OhdsiRWebAPI R pacakge approach to export-import from old-to-new WebAPI instances.

Hi again Chris,

Thank you for all that information and apologies for the late reply.

I attached the relevant errors from the catalina.out file that gets created when tomcat is started with the webapi. It seems that there is an error in the migration scripts here is the relevant line:
Location : db/migration/postgresql/V2.8.0.20200109100200__cohort_sample_tables.sql (/opt/apache-tomcat-8.5.71/webapps/WebAPI/WEB-INF/classes/db/migration/postgresql/V2.8.0.20200109100200__cohort_sample_tables.sql)
Line : 3
Statement : CREATE TABLE webapi.cohort_sample(
id INTEGER PRIMARY KEY NOT NULL,
name VARCHAR(255) NOT NULL,
cohort_definition_id INTEGER NOT NULL,
source_id INTEGER NOT NULL,
size INTEGER NOT NULL,
age_min SMALLINT NULL,
age_max SMALLINT NULL,
age_mode VARCHAR(24),
gender_concept_ids VARCHAR(255) NULL,
created_by_id INTEGER,
created_date TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT (now()),
modified_by_id INTEGER,
modified_date TIMESTAMP WITH TIME ZONE,
CONSTRAINT fk_cohort_sample_definition_id FOREIGN KEY (cohort_definition_id)
REFERENCES webapi.cohort_definition (id) ON DELETE CASCADE,
CONSTRAINT fk_cohort_sample_source_id FOREIGN KEY (source_id)
REFERENCES webapi.source (source_id) ON DELETE CASCADE
)
Atlas_Migration_Error.pdf (114.8 KB)

As shown we are using postgresql to hold our instance of the ohdsi data base. As far as I know there were never any manual changes to the database but if the error tells you anything else please let me know if there are places that things can be changed to be correct.

Thank you for the info on the R tool I will look into using that as well.

Best,
Alex

You didn’t include the most important part of the error message, but I found it in the attached log:

SQL State : 42830
Error Code : 0
Message : ERROR: there is no unique constraint matching given keys for referenced
table "source"

So, it’s saying that there is no unique constraint matchign the given keys for table ‘source’. I wasn’t aware of this requirement for postgresql, but the foreign key to the source talbe is source_id, so it appears that in order to set a CONSTRAINT for FOREIGN KEY, we need a unique constraint (ie: a primary key) on the webapi.source(source_id) column. I checked my local env, and source_id is the primary key of the source table, and it appears that this table has been set this way since version 1.0: https://github.com/OHDSI/WebAPI/blob/master/src/main/resources/db/migration/postgresql/V1.0.0.7.0__sources.sql.sql.

Got it thank you for clarifying!
So would it be as simple as adding that primary key to the source table?

ie. adding a primary key named pk_source with a column of source_id?

Would this cause any problems to the existing running version of Atlas?

Thanks again for your help here.

Best,
Alex

You can give it a try, but my only concern is that if there should have been a PK on the source table but there isn’t, I’m not sure what else may be missing from your database schema.

Yeah so it cleared up the first error but encountered the next one:

Migration V2.8.0.20201022120031__concept_ancestor_and_descendants.sql failed

SQL State : 23505
Error Code : 0
Message : ERROR: duplicate key value violates unique constraint “permission_unique”
Detail: Key (value)=(vocabulary:pdbv36:concept::ancestorAndDescendant:get) already exists.
Location : db/migration/postgresql/V2.8.0.20201022120031__concept_ancestor_and_descendants.sql (/opt/apache-tomcat-8.5.71/webapps/WebAPI/WEB-INF/classes/db/migration/postgresql/V2.8.0.2020102212003$
Line : 3
Statement : INSERT INTO webapi.sec_permission (id, value, for_role_id)
SELECT nextval(‘webapi.sec_permission_id_seq’),
REPLACE('vocabulary:%s:concept:
:ancestorAndDescendant:get’, ‘%s’, REPLACE(REPLACE(value, ‘source:’, ‘’), ‘:access’, ‘’)),
role_id
FROM webapi.sec_permission sp
JOIN webapi.sec_role_permission srp on sp.id = srp.permission_id
WHERE sp.value LIKE ‘source:%:access’

Though when looking into the table there doesn’t seem to be a row with the specified value unless I am misunderstanding how it is supposed to work.

I imagine it might be like this for a while though this one seems to be an issue of the permission setting already existing when trying to insert it into the table.

It may make more sense to go with the ROhdsiWebApi route thoughts?

I think you’re getting close, so it may be worth continuing down this path to just address these migration issues. The current migration error is that the v2.8 release wanted to re-use the source permission to the ‘access’ permission, so they did a trick where they read for permissions that look like ‘source:%:access’ (% is a wildcard) and insert those rows back after replacing source and access to ‘’ (empty string) as the new permission name. I think this was to deal with existing permissions on sources, but you may have uncovered a corner case where it doeesn’t work.

If you could run the following queries just so we can understand what’s in your data and how we might account for it:

SELECT sp.value as from_value,
REPLACE('vocabulary:%s:concept::ancestorAndDescendant:get’, ‘%s’, REPLACE(REPLACE(value, ‘source:’, ‘’), ‘:access’, ‘’)) as ,to_value,
role_id
FROM webapi.sec_permission sp
JOIN webapi.sec_role_permission srp on sp.id = srp.permission_id
WHERE sp.value LIKE ‘source:%:access’

This is the query the migration script is attempting to run that is going to insert new permission records into sec_permission. If you can check to see if any of the resulting 'to_value records already exist in the sec_rpermission table, and that will explain why the migration failed: the logic is tryign to insert the same permission record twice (ie: sec_permission.value must be unique).

I’m not totally familiar with the REPLACE() with the %s syntax, but if you execute the above query, at least we can understand why the REPLACE() is leading to a duplicate key in the sec_permission table.

Update: the %s isn’t a special syntax, but rather the %s is being used as a ‘placeholder’ to replace the %s with the value with the value from REPLACE(REPLACE(value, ‘source:’, ‘’), ‘:access’, ‘’). So that explains the %s (it’s just a trick they used to start with a string template and then replace it with the final form), so I still need you to run the above query and tell me if any of the yielded ‘values’ match any of the existing values in your sec_permission table.

Okay that sounds good to me, attached is the results of the query running.

Screen Shot 2022-08-17 at 3.02.02 PM

The values all seem to be related to the access to our CDM data sources, these change every month to every quarter as new data refreshes happen would this be related to any issues we are seeing?

On a separate note our current production instance has seemingly gone done and is giving the following error message:
2022-08-17 13:48:15.490 DEBUG taskExecutor-4 org.springframework.jdbc.datasource.DriverManagerDataSource - - Creating new JDBC DriverManager Connection to [jdbc:postgresql://redcated:5432/ohdsi?ssl=true&sslmode=require]
2022-08-17 13:48:15.515 DEBUG localhost-startStop-1 org.springframework.orm.jpa.JpaTransactionManager - - Exposing JPA transaction as JDBC transaction [org.springframework.orm.jpa.vendor.HibernateJpaDialect$HibernateConnectionHandle@29c1643f]
Hibernate: select cohortgene0_.id as id1_10_, cohortgene0_.source_id as source_i2_10_, cohortgene0_.execution_duration as executio3_10_, cohortgene0_.fail_message as fail_mes4_10_, cohortgene0_.include_features as include_5_10_, cohortgene0_.is_valid as is_valid6_10_, cohortgene0_.person_count as person_c7_10_, cohortgene0_.record_count as record_c8_10_, cohortgene0_.start_time as start_ti9_10_, cohortgene0_.status as status10_10_ from webapi.cohort_generation_info cohortgene0_ where cohortgene0_.status in (? , ?)
2022-08-17 13:48:15.543 WARN localhost-startStop-1 org.hibernate.engine.jdbc.spi.SqlExceptionHelper - - SQL Error: 0, SQLState: 42703
2022-08-17 13:48:15.543 ERROR localhost-startStop-1 org.hibernate.engine.jdbc.spi.SqlExceptionHelper - - ERROR: column cohortgene0_.include_features does not exist
Position: 171
2022-08-17 13:48:15.546 DEBUG localhost-startStop-1 org.springframework.orm.jpa.JpaTransactionManager - - Initiating transaction rollback
2022-08-17 13:48:15.546 DEBUG localhost-startStop-1 org.springframework.orm.jpa.JpaTransactionManager - - Rolling back JPA transaction on EntityManager [org.hibernate.jpa.internal.EntityManagerImpl@7cc6e8a4]
2022-08-17 13:48:15.554 DEBUG localhost-startStop-1 org.springframework.orm.jpa.JpaTransactionManager - - Closing JPA EntityManager [org.hibernate.jpa.internal.EntityManagerImpl@7cc6e8a4] after transaction
2022-08-17 13:48:15.554 DEBUG localhost-startStop-1 org.springframework.orm.jpa.EntityManagerFactoryUtils - - Closing JPA EntityManager
2022-08-17 13:48:15.555 WARN localhost-startStop-1 org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext - - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘cohortAnalysisService’: Unsatisfied dependency expressed through field ‘definitionService’; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘cohortDefinitionService’: Invocation of init method failed; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not extract ResultSet
2022-08-17 13:48:15.555 INFO localhost-startStop-1 org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - - Shutting down ExecutorService ‘taskExecutor’
2022-08-17 13:48:15.567 DEBUG taskExecutor-4 org.springframework.orm.jpa.JpaTransactionManager - - Exposing JPA transaction as JDBC transaction [org.springframework.orm.jpa.vendor.HibernateJpaDialect$HibernateConnectionHandle@4a73545a]
2022-08-17 13:48:15.574 DEBUG taskExecutor-4 org.springframework.jdbc.core.JdbcTemplate - - Executing prepared SQL update
2022-08-17 13:48:15.574 DEBUG taskExecutor-4 org.springframework.jdbc.core.JdbcTemplate - - Executing prepared SQL statement [UPDATE webapi.BATCH_STEP_EXECUTION set START_TIME = ?, END_TIME = ?, STATUS = ?, COMMIT_COUNT = ?, READ_COUNT = ?, FILTER_COUNT = ?, WRITE_COUNT = ?, EXIT_CODE = ?, EXIT_MESSAGE = ?, VERSION = ?, READ_SKIP_COUNT = ?, PROCESS_SKIP_COUNT = ?, WRITE_SKIP_COUNT = ?, ROLLBACK_COUNT = ?, LAST_UPDATED = ? where STEP_EXECUTION_ID = ? and VERSION = ?]
2022-08-17 13:48:15.586 DEBUG taskExecutor-4 org.springframework.jdbc.core.JdbcTemplate - - SQL update affected 1 rows
2022-08-17 13:48:15.586 DEBUG taskExecutor-4 org.springframework.jdbc.core.JdbcTemplate - - Executing prepared SQL query
2022-08-17 13:48:15.586 DEBUG taskExecutor-4 org.springframework.jdbc.core.JdbcTemplate - - Executing prepared SQL statement [SELECT VERSION FROM webapi.BATCH_JOB_EXECUTION WHERE JOB_EXECUTION_ID=?]
2022-08-17 13:48:15.595 DEBUG taskExecutor-4 org.springframework.orm.jpa.JpaTransactionManager - - Initiating transaction commit
2022-08-17 13:48:15.595 DEBUG taskExecutor-4 org.springframework.orm.jpa.JpaTransactionManager - - Committing JPA transaction on EntityManager [org.hibernate.jpa.internal.EntityManagerImpl@bed84c5]
2022-08-17 13:48:15.608 DEBUG taskExecutor-4 org.springframework.orm.jpa.JpaTransactionManager - - Closing JPA EntityManager [org.hibernate.jpa.internal.EntityManagerImpl@bed84c5] after transaction
2022-08-17 13:48:15.608 DEBUG taskExecutor-4 org.springframework.orm.jpa.EntityManagerFactoryUtils - - Closing JPA EntityManager
2022-08-17 13:48:15.608 DEBUG taskExecutor-4 org.springframework.orm.jpa.JpaTransactionManager - - Creating new transaction with name [org.springframework.batch.core.repository.support.SimpleJobRepository.updateExecutionContext]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
2022-08-17 13:48:15.611 ERROR taskExecutor-4 org.springframework.batch.core.step.AbstractStep - - Encountered an error executing step warmCacheStep in job warmCache
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is java.lang.IllegalStateException: EntityManagerFactory is closed

Could this be related to the migration of the database causing the old version to fail?

Thanks again for all the help so far its greatly appreciated.

Yes. Part of the 2.8 version was to drop the column, so when you placed the old version back on the updated database, the old version was looking for columns that were removed. If you have a backup, you can restore to that. Or, we can press ahead and understand why the migration failed:

From your query results, we see how it was using the existing permissions for :access to create permissions for :ancestorAndDescendant. If you notice from your output, we have duplicate permissions: source:pdbv36:access appears multiple times (and cov_wk2): role_id 1028, 1007 adn 1008. This, in turn, leads to duplicate inserts into your sec_permission table, which is why the migration fails (for good reason). I’m not sure how those duplicate records were inserted into that table when there was a UNIQUE constraint on that table since Feb, 2018.

I’ve done some digging on this, and the issue is because the migration wasn’t tested to work when you have multiple roles given the same permission. The way the migration was implemented conflicts with the UNIQUE constraint on the sec_permission table…and the way it should have been implemetned was to create the sec_permission rows into a temp table, insert the unique permissions back into the sec_permission table and then use the same temp table for the role-assignments. None of this is your fault, but to work around this will be a little technical challenge for you:

I’m going to post up an alternative migration script that you will need to replace the contents of the existing migration script ( db/migration/postgresql/V2.8.0.20201022120031__concept_ancestor_and_descendants.sql) with the content I will give you in my next forum post (I need to write it), and this will mean that you will need to rebuild the WebAPI.war file and then re-deploy the application to your production enviornment. Once we know it works for you, we can push the fix as a hotfix release. I’m sorry to make you the test subject for this, but hopefully it will get you up and running. Note: we can’t just drop the UNIQUE index on the sec_permission table, it will pollute the permissiont able with duplicate records.

I’ll reply soon with an updated script.

-Chris

Ok, this is the updated migration script content that I tested on a local dev instance of Postgresql which I have 95% confidence that you can run in your own environment.

Replace the contents of the file: db/migration/postgresql/V2.8.0.20201022120031__concept_ancestor_and_descendants.sql

with the following:

CREATE TEMP TABLE temp_migration (
  from_perm_id int,
  new_value character varying(255)
);

INSERT INTO temp_migration (from_perm_id, new_value)
SELECT sp.id as from_id,
  REPLACE('vocabulary:%s:concept:*:ancestorAndDescendant:get', '%s', REPLACE(REPLACE(value, 'source:', ''), ':access', '')) as new_value
FROM ${ohdsiSchema}.sec_permission sp
WHERE sp.value LIKE 'source:%:access';

INSERT INTO ${ohdsiSchema}.sec_permission (id, value)
SELECT nextval('${ohdsiSchema}.sec_permission_id_seq'), new_value
FROM temp_migration;

INSERT INTO ${ohdsiSchema}.sec_role_permission (id,role_id, permission_id)
SELECT nextval('${ohdsiSchema}.sec_role_permission_sequence'),
  srp.role_id,
  sp.id as permission_id
FROM temp_migration m
JOIN ${ohdsiSchema}.sec_permission sp on m.new_value = sp.value
JOIN ${ohdsiSchema}.sec_role_permission srp on m.from_perm_id = srp.permission_id;

This is what it is doing:

First, it creates a temp_migration temp table to store the conversion of existing permissions that match source:%:access and stores the original permission_id for source access, and generates a new permission string based on the REPLACE. We are doing this so that we can take all the existing sources you’ve created permissions for, and introduce a new permission that looks like ancestorAndDescendant:get.

Next, we’re inserting new sec_permission records by getting a new permission_id (via the sec_role_permission_sequence) and using the generated ancestorAndDescendant permission for each source.

Finally, we need to assign the same roles that are assigned to the permission for source:%:access. So, we do a 3-way join on temp_migration (to get the map between the source:%:access permission id, and the new permission value), a join to sec_permission to get the new permission_id for the new ancestorAndDescendant permission, and a join to sec_role_permission so we can find all the roles that were assigned to the source:%:access so we can copy those role assignments to the new ancestorAndDescendant permission.

Whew! Hope that all made sense, but I would appreciate it if you could replace the above migration script with your existing one, then rebuild WebAPI and try to migrate again. If it works, we’ll take this updated migration script and release a hotfix.

We may get new errors with later migrations, but I believe this will get us past this migration step.

-Chris

Hey Chris,

That makes sense to me! I will go in and give the replacement a try and see if it fixes the issue. I will keep you update with the progress.

Thank you so much for working on this and getting a script out to me its really appreciated!!

Best,
Alex

Hi Chris,

So the script worked like a charm and the migration seemed to move past the part it was hanging on earlier and even seems to say it made it thru all the migration steps.
However, there is another error that it is encountering and I’m really sure what is causing it.
From what I can gather the relevant message is as follows:
2022-08-18 11:14:42.905 ERROR localhost-startStop-1 org.hibernate.engine.jdbc.spi.SqlExceptionHelper - - ERROR: operator does not exist: integer = character varying
Hint: No operator matches the given name and argument types. You might need to add explicit type casts.
Position: 404

This doesn’t seem very informative.

I also attached the full (relevant) part of catalina.out from when I restarted tomcat with the new webapi.war file

Thanks again as always

Best,
Alex

That looks like someting related to performing a sql query and doing a where clause where col1 = col2 but col1 is integer and col2 is varchar. usually along with that error you’ll see the query that caused the error, so if you could find the SQL that precedes that error, that would be helpful.

Gotcha yeah so looking at the rest of the log there doesn’t seem to be any place where the query is written out. I forgot to attach the catalina.out but I attached it here.

catalina_current.out.pdf (175.7 KB)

Looking into the postgres log did get me what seems to be the query that causes the problem:

2022-08-18 15:52:01 EDT [62fe9861.f680:6]: 10.44.1.78 ohdsi_admin_user ohdsi: STATEMENT: select executioni0_.analysis_id as analysis1_45_, executioni0_.source_id as source_i2_45_, executioni0_.execution_duration as executio3_45_, executioni0_.is_canceled as is_cance4_45_, executioni0_.is_valid as is_valid5_45_, executioni0_.message as message6_45_, executioni0_.start_time as start_ti7_45_, executioni0_.status as status8_45_ from webapi.ir_execution executioni0_ where executioni0_.status in ($1 , $2)

I honestly can’t make heads nor tails of this one so any recommendations you may have would be super useful.

Thanks as always for your help with this.

Best,
Alex

Ok, I went through the logs and found a line number that corresponds to the error, and it is this method:

  private void invalidateIRExecutions() {

    getTransactionTemplateRequiresNew().execute(status -> {

      List<ExecutionInfo> executions = irExecutionInfoRepository.findByStatusIn(INVALIDATE_STATUSES);
      invalidateExecutions(executions);
      irExecutionInfoRepository.save(executions);
      return null;
    });
  }

This code looks like it would produce the query you mention because when WebAPI starts up it wants to find any job running in PENDING or RUNNING state. That’s the last part: where executioni0_.status in ($1 , $2). The issue seems to be that these two states are being either converted to a string or integer (not sure which) but the underlying column is the opposite (integer or string, not sure which).

In my environment, i am showing the ‘status’ column of ir_execution as status character varying(128). Can you confirm the column type in your schema?

Some additional information: at one point, the status column was an ‘int’ and I’m wondering now if you are running a version of WebAPI that assumes that the column is an INT, but you ran migrations that expects the column to be varchar, that would be an issue.

Can you confirmt he following:

  1. What version of WebAPI are you running?
  2. What is the type of ‘status’ in the table ir_execution?
  3. What are the values in the ‘status’ table (select distinct status from ir_execution)
  4. If you can look into the source that you compiled into the WebAPI.war, can you tell me what’s in the file ExecutionInfo.java for the field ‘status’, it should look like:
  @Column(name="status")
  @Enumerated(EnumType.STRING)
  private GenerationStatus status;    

-Chris
,

Hi Chris,

  1. The version of the WebAPI 2.9.0 is as is shown in the log here:
    Current version of schema “webapi”: 2.9.0.20210812164224

  2. Status is an integer

  3. There is actually no data in the ir_execution table at all.

  4. @Column(name="status") @Enumerated(EnumType.STRING) private GenerationStatus status;
    That seems to look the same as above.

Since there is nothing in the ir_execution table would it make sense to just change the coloumn to be varchar?

Best,
Alex

t