CPT4 Concept Processing Issue

rbajr · December 8, 2022, 3:11pm

After executing the cpt4.jar utility to update CONCEPTS, I observe the following output:

[INFO ] 2022-12-07 09:53:27.411 [main] ConceptService - Writing updated data to CONCEPT.csv
Updated CPT4 records: 16632/16632
[INFO ] 2022-12-07 10:27:15.867 [main] ConceptService - Not processed cpt4 concepts: 12487.
See logs/not-processed-concepts-12-07-2022-09-53-25.out, file.
You can find more information about errors in the logs/logfile.log file

I have run the cpt4.jar utility numerous times, including on different machines, with the same result. A colleague also ran the utility this morning, with the same result.

This issue seems similar to a previous issue posted on this forum: CPT4 cpt.bat process issue.

Seems like a possible issue with the java code api calls to UMLS.
Any insights/solutions would be appreciated.

solmaz.eradat · December 8, 2022, 3:35pm

Hi @rbajr

We were actually running this yesterday for our project and had the same issue.

A workaround that seemed to work for us was, if you keep re-running the java command in the same terminal session it seems to update and work

Thanks,
Solmaz

rbajr · December 8, 2022, 3:39pm

@solmaz.eradat thanks. I will give that a try.

Interestingly when my colleague ran the process he received the following error after the process had completed. On my runs I didn’t see any exceptions (including in the .log, .out files):

Exception in thread “main” org.odhsi.utils.cpt.Cpt4Exception: Unexpected code Response{protocol=h2, code=401, message=, url=https://utslogin.nlm.nih.gov//cas/v1/tickets/TGT-93158-1emw14dvlbf4e4lf-cas}
at org.odhsi.utils.cpt.restclient.RestTicketClient.checkSuccessful(RestTicketClient.java:76)
at org.odhsi.utils.cpt.restclient.RestTicketClient.logout(RestTicketClient.java:113)
at org.odhsi.utils.cpt.restclient.RestTicketClient.close(RestTicketClient.java:125)
at org.odhsi.utils.cpt.Application.main(Application.java:49)

solmaz.eradat · December 8, 2022, 3:51pm

Ran the command on a window machine with the current version

Thomas_White · December 8, 2022, 7:44pm

I’m having the same problem. For me, there are always significant # not processed, even when run cpt.sh multiple times in the same session (or starting from scratch). The first time, it was 12488 (and reduced that to 3625 not processed when ran it a second time. Then, when I started from scratch, it was 12450 not processed even when ran multiple times.

However, when I review the logs/not-processed-concepts.out file, they all appear to be valid CPT codes, and I get get accurate labels when searching for them via athena.ohdsi.org

shinjinaka · December 9, 2022, 8:03pm

I’m having the same problem. I tried to walk around this issue by using CPT4 concept that I previously downloaded in Sep 2022 but it failed.
SQL Error [23503]: ERROR: insert or update on table “concept_relationship” violates foreign key constraint “fpk_concept_relationship_c_1”
Detail: Key (concept_id_1)=(777313) is not present in table “concept”.
Looking at the Athena last update for CPT4, it shows “01-May-2022”, but Athena has Valid start “12-Oct-2022”. so we have new CPT4 version in Athena? @mik @Dymshyts Could you fix the issue like you did last year? Thanks.

Mikita_Salavei · December 12, 2022, 8:34am

This problem was also discussed some days ago on GitHub here

david_seligman · January 25, 2023, 7:36pm

Our engineers have run into the same problem. I can’t tell from the threads if there’s a fix available or whether we should use old tables instead. @Christian_Reich?

mik · January 25, 2023, 11:28pm

Hi @david_seligman , do you reference to " violates foreign key "? If that is the case, you are trying to mix vocabulary versions in your tables. When loading vocabularies clear your vocabulary tables first and load them fresh…
If you are experiencing issues with operating the cpt4.jar, we have just noticed that we need to finetune that again for the latest release. Bear with us a little.

david_seligman · January 26, 2023, 2:34pm

@mik Thank you. This is the error we got “2022-12-06 00:18:10.517 [main] ConceptService - Not processed cpt4 concepts: 12714. See logs/not-processed-concepts-12-05-2022-22-13-43.out, file.” We can post the log file if it would be helpful.

mik · January 26, 2023, 5:00pm

okay, @david_seligman , sounds like you need the fixed cpt4.jar. We are currently testing it and I will post here, once it is available through ATHENA. You will have to re-initiate the vocabulary download (go to the download page, select your vocabs and request the download). My assumption is that it should be available tomorrow (stay tuned).

david_seligman · January 26, 2023, 5:18pm

@mik Thank you! We’ll wait to re-try our download until then.

Konstantin_Yaroshove · January 28, 2023, 7:01pm

There are some obstacles that will need more time to address them. The tool will be updated on Monday.

Konstantin_Yaroshove · January 30, 2023, 2:05pm

The new version of CPT4 tool was published in ATHENA.

solmaz.eradat · January 31, 2023, 4:17pm

Thank @Konstantin_Yaroshove for the update. I tested rerunning the process after the new release and did not encounter any issues.

I found the update took some time to do (around 4-5 hours). Is this expected?

Many thanks,
Solmaz

Konstantin_Yaroshove · January 31, 2023, 6:52pm

@solmaz.eradat Yes, it’s expected. Unfortunately, this limitation comes from UMLS side.
We are looking for possible solutions to improve the speed.

solmaz.eradat · February 8, 2023, 12:08pm

Hi @Konstantin_Yaroshove .

After the recent CPT4 update, we were updating our vocabulary tables using the scripts in the DDL folder https://github.com/OHDSI/CommonDataModel/blob/main/inst/ddl/5.4/sql_server/OMOPCDM_sql_server_5.4_constraints.sql.

An error was generated when running line 319 https://github.com/OHDSI/CommonDataModel/blob/main/inst/ddl/5.4/sql_server/OMOPCDM_sql_server_5.4_constraints.sql#L319

It seems the following concept_ids:

45889325
45890303

are not in the concept table.

Checked on the Athena website and these ids are coming from the CPT4 vocab.

The fields and tables below are the once be effected:

many thanks,
Solmaz

mik · February 8, 2023, 1:34pm

Hi @solmaz.eradat ,
I checked my csv files and found the concept id 45889325 and 45890303 in both, the CONCEPT_CPT4 and the CONCEPT.csv file after reconstitution. As tiresome it is, can you re-run the download and reconstitution exercise if you cannot find those id’s in your csv files? If you do find them, there must be something happening during the import.
cheers - mik

solmaz.eradat · February 13, 2023, 1:51pm

Hi @mik . Thanks for the suggestion. Reran the steps again with a fresh Athena download and cpt run and no issues this time. Thanks.

rnath · December 7, 2023, 9:55pm

I am facing the same issue with the latest cpt4.jar that comes with the vocabulary download where a large no of concept update is failing. Tried looking for the source code but couldn’t find it. By any chance, is the cpt4.jar source code available on Ohdsi git?