Not processed cpt4 concepts: 13553

[Jeff_Stroup]

Hi,

I’m getting the following message after processing CPT4 concepts

[INFO ] 2024-09-18 22:40:41.626 [main] ConceptService - Not processed cpt4 concepts: 13553. See logs/not-processed-concepts-09-18-2024-21-48-53.out, file. You can find more information about errors in the logs/logfile.log file

I’m on a PC so I have no /logs folder and no log folder was created in the current folder. I scrolled through all of this output and no individual items showed any errors.

Updated CPT4 records: 17451/17455
Updated CPT4 records: 17452/17455
Updated CPT4 records: 17453/17455
Updated CPT4 records: 17454/17455
Updated CPT4 records: 17455/17455

This post alludes to a similar issue which was ostensibly fixed.

Latest Vocabulary Files - CPT4 processing issues

Hello, We have several sites all experiencing issues processing the new concept.csv file. We have all run the process multiple times and still cannot get past 283 concepts not processed. I looked at the concepts directly in UMLS and noticed that for some that are not processing, the UMLS vocabulary is ‘HCPT’ whereas the .OUT file shows a vocabulary of ‘CPT’. Any recommendations would be greatly appreciated. Best, Piper

Can someone please identify even one CPT4 concept that is supposed to be present but is not? My only other approach is to compare CPT4 concepts from the previous vocab version to locate a gaping 13553-concept hole. Otherwise, I don’t see a clearly defined problem yet.

Thanks,
Jeff

[Jeff_Stroup]

OK, I did confirm a massive difference in CPT4 codes between the new an previous versions. The difference did not add up to 13553 but maybe that accounts for new CPT4 codes?

   -- old version CPT4 count
select
	count(1)
from
	vocab.concept
where
	vocabulary_id = 'CPT4'

16,632

-- new version CPT4 count
select
	count(1)
from
	vocab.concept_v5_0_30_aug_24
where
	vocabulary_id = 'CPT4'
	
3,902


select 16632 - 3902
12,730

[DTorok (Don Torok)]

Sounds like you might be getting a little frustrated. I am curious, what does “I’m on a PC so I have no /logs folder” mean? That is, why would being on a PC affect whether a logs folder is created.

[Jeff_Stroup]

Often in Linux Land you can simply go to a /log folder off the root. I guess it was just another way of saying “I can’t find the log file”.

[DTorok (Don Torok)]

The log file folder should be a sub directory of whatever directory you ran cpt4.jar. Sure it is not there?

[Alexdavv (Alexander Davydov)]

Hi @Jeff_Stroup

You should not be getting any negative delta - all the not processed concepts should be processed, and therefore, added to the concept table.
Try to rerun the cpt4 tool a couple of times, otherwise think of the connection/access issues that may persist in your setup.

[Jeff_Stroup]

@DTorok I was looking in the folder that the entire automated vocabulary install starts in. My apologies. They do at least tell me what concepts failed so I guess that’s somewhat useful.

[INFO ] 2024-09-18 21:55:28.304 [pool-2-thread-10] ConceptUpdater - Update status:FAIL. The code value for id 2314327 is []

[DTorok (Don Torok)]

OK, now follow Alexander’s advice and run the cpt4 jar again. DO NOT try to clean up anything. The cpt4.jar is smart enough to know what concepts did not get processed. Some people have had to run cpt4.jar for multiple time until it finally says that everything has been processed.

[Jeff_Stroup]

Thanks. I’ll continue to re-try.I tried again (manually this time) and got a different number of unprocessed cpt4 concepts…

Not processed cpt4 concepts: 13581

I believe my UMLS license needed to be renewed so I think I successfully renewed it since the download link finally delivered a zip file with contents instead of a zero byte file.
Since the command line of the CPT jar requires an API key I grabbed the key from my profile. I guess it fails regardless so I’ll keep trying - and I won’t try to clean up the CONCEPT.CSV file this time!

[Jeff_Stroup]

We finally got the CONCEPT table populated with all of the required CPT4 codes and in only two tries. @DTorok’s advice to not clean up anything was key. Thanks! But why does the CPT4 script even require subsequent passes in the first place? It breaks our automated process (which expects a single run of the script).

To remain fully automated we would have to add logic to keep re-running the CPT4 script until it had no more errors. Not the end of the world obviously but this is the view from my world.

One additional noteworthy concern is that our automated process has to crack open the VOCABULARY.CSV file and then convert it to a table just to get the version number of the vocabulary. Wouldn’t it be splendid if the version was somehow included in the name of the delivered zip file?!!

Thanks,
Jeff