ATLAS-Search-RC and DRC are populated as 0

CSC · April 27, 2020, 3:53pm

We are using the ‘Search’ tab in ATLAS and trying to look for ‘hypertension’ related concepts.
We get around 1300+ results for hypertension, but the RC and DRC counts are 0 (zero) for all of them.

We checked CONDITION_OCCURRENCE table in our CDM for few of them (the ones with Domain=‘Condition’ and Vocabulary=‘ICD9CM’) and got row count greater than 0 (infact in the range of 100k rows).
We have been trying to find the root cause/source of this.
We tried to check the activity on the database to identify the query being hit (PostgreSQL) to retrieve this result but don’t see any query.

Can you please help us in identifying the source of the values that get populated under RC and DRC columns?

Thanks in Advance.

Chris_Knoll · April 27, 2020, 4:01pm

@AnthonyMolinaro may have better insight to those columns, since the behavior has changed since the initial introduction of those columns in the UI. My current understanding is that those statistics come form the Achilles results But Anthony may have more up-to-date information on that.

As far as finding ICD9 concepts in the CONDITION_OCCURRENCE table, which column are you looking at: SOURCE_CONCEPT or CONDITION_CONCEPT? If you’re finding them in the CONDITION_CONCEPT_ID column, then I think you have an ETL problem because those ICD9’s should have been mapped over to the standard concepts (found in SNOMED).

CSC · April 27, 2020, 4:09pm

Hello @Chris_Knoll,

Thank you for the prompt reply.

I am looking for the concept-ids in CONDITION_SOURCE_CONCEPT_ID column of CONDITION_OCCURRENCE. Along with this, I am also checking on CONDITION_SOURCE_VALUE for the concept codes.

Thanks.

Chris_Knoll · April 27, 2020, 5:21pm

Ok, thanks for confirming. Initially, we were only showing record counts for standard concepts in the output, however, my understanding is that recent changes have looked at both the standard concept field (condition_concept_id) and the non-standard concept field (condition_source_concept_id), although I’m not exactly certain where those changes were made (if it’s a new version of Achilles, or if it is some additional script that does that).

@AnthonyMolinaro: do you have this information, or @Ajit_Londhe?

Ajit_Londhe · April 28, 2020, 4:29am

Achilles does have analyses that characterize source_concept_id fields. For example, analysis id 425, counts condition_source_concept_id values.

CSC · April 28, 2020, 12:56pm

Hello @Ajit_Londhe,

We are using Atlas version 2.7.4 with WebAPI version 2.7.4.

We indeed had run Achilles analysis. I checked for analysis-id 425 and for condition_source_concept_id=‘44834715’ in achilles_results table in results schema and found a non-zero value. Below is the screenshot -

I checked for the SQL script (from here)that is used for analysis-id 425 and executed against my database.Below is the screenshot-

I am still unsure why the values are still 0(zero) when being displayed in Atlas
Please help.

Thanks in advance.

Ajit_Londhe · April 28, 2020, 1:52pm

Hmm. Do you have the concept_hierarchy table?

CSC · April 28, 2020, 1:56pm

Hello @Ajit_Londhe,

Yes, it’s there under results schema.

CSC · April 29, 2020, 6:05am

Hello @Ajit_Londhe, @Chris_Knoll,

We did some more analysis and executed few more queries on the database to ensure that we have proper data in the tables.
We executed the query of getConceptRecordCount on our database and it returned a non-zero count.

We also tried refreshing cache for the data source from the Configuration tab/page of Atlas and executed the loadConceptRecordCountCache query to ensure that non-zero counts are present. But still the count on Atlas is being displayed as 0 (zero).

While refreshing the Atlas->Search page we found that the call through WebAPI to URL https://servername/WebAPI/cdmresults/omop_cdm/conceptRecordCount doesn’t return any response.

Can you please help us in identifying the cause of this issue?

Thanks.

Ajit_Londhe · April 29, 2020, 4:02am

Has the WebAPI server been restarted since you finished running Achilles? Perhaps the vocab cache in WebAPI hasn’t been refreshed.

CSC · April 29, 2020, 6:15am

Hello @Ajit_Londhe,
We had restarted Tomcat service but not the physical server.
Thanks.

Ajit_Londhe · April 29, 2020, 2:06pm

Oh that’s fine then. I was just referring to Tomcat.

Ajit_Londhe · April 29, 2020, 2:49pm

@Chris_Knoll – I’m at a loss here. From an Achilles perspective, nothing has changed recently with any of the codebase. I’ve recently run it and added a CDM to Atlas and all the record counts come through.

@CSC – have you set up your source correctly in WebAPI? That is, does the source daimon for the results have a priority of 1?

CSC · April 29, 2020, 3:40pm

Hello @Ajit_Londhe, @Chris_Knoll
The priority was set to 2 for results in webapi’s source_daimon table.
I changed it to 1, logged-in to Atlas again and performed the search for ‘hypertension’ but still zero count under RC and DRC.
Should I 'Refresh Cache’ for the data source from the Configuration page in Atlas?

Thanks.

Ajit_Londhe · April 29, 2020, 3:54pm

I would restart Tomcat, then in Atlas, refresh the cache in config.

Chris_Knoll · April 29, 2020, 4:06pm

I’m not sure either. The getConceptRecordCount query is the query that is executed at the conceptRecordCount endpoint. However, that endpiont is a POST, so you can’t just go to the URL and look for a result (it should give you something like ‘not allowed’).

When I search on my local system, and look at the console, I see the request, and the payload:

The response is shwoing non-zero values:

My only hypothesis is that the source configuration is looking at the wrong schema and the wrong schema just has empty tables in it. I can’t understand why you’d get results out of a raw sql query to the DB, but the server’s doing the same thing (supposedly) and getting nothing…