OHDSI Home | Forums | Wiki | Github

Error running Achilles against Databricks

I am trying to run Achilles against CDM in Databricks using RStudio. I am seeing an error -

2023-03-22 00:00:07	Beginning single-threaded execution
Connecting using Spark JDBC driver
  |=================================================================================| 100%
Executing SQL took 4.42 secs
Error in rJava::.jcall("java/lang/System", "S", "getProperty", "java.version") : 
  java.sql.SQLException: [Simba][JDBC](10040) Cannot use commit while Connection is in auto-commit mode.

In the JDBC connection string I am setting the following parameters (based on information from other threads) - UseNativeQuery=1;IgnoreTransactions=1. (I also tried without the IgnoreTransactions=1, I am getting the same error).

Any hints here would be helpful.

This is a known issue with DatabaseConnector::insertTable (Achilles makes use of this to insert static analysis information).

The Databricks Working Group is looking into this, but so far, the autocommit capability needed isn’t looking promising.

Thanks for the clarification. Just wanted to check if there any workarounds that can be used to run Achilles against CDM in Databricks using Spark JDBC ?

As a workaround I ran Achilles with sqlOnly = TRUE, via Rstudio, which generated an achilles.sql file. I ran the generated achilles.sql against Databricks. The script ran fine, and I can see tables like achilles_results, achilles_results_dist, with data populated, but looks like the ACHILLES_analysis table is not created as a part of the SQL script (not sure if there are other tables missing as well).

Consequently, when I try to browse the data source dashboards in Atlas this fails as the ACHILLES_analysis table is missing.

Are there any other scripts that need to be run to make sure all the Achilles tables are created in the results schema ?

You can simply take the achilles_analysis CSV file from here and create it as a table

1 Like

Thanks @Ajit_Londhe ! This did the trick. Now I am able to view the Atlas data source dashboards.

1 Like