I’m happy to announce that we’re getting closer to official Databricks (Spark) support in much of the OHDSI stack! We will present about this a bit during the August 17, 2021 Community call.
Here’s a roadmap of upcoming deliverables:
Add Spark translation support to SqlRender
ETA: Master branch in August 2021
We now have Spark support in the develop branch of SqlRender. We’ve provided a testing Databricks cluster to the OHDSI CI platform, but some configuration is pending.
Add Spark support to Atlas / WebAPI
ETA: Atlas v2.10 end of August 2021
Thanks to @gregk and the Odysseus team, we have a working version of Atlas with Spark support. We are aiming for submitting a PR for these changes in August, targeting Atlas v2.10.
Add Spark support for DatabaseConnector
ETA: September 2021
Checking with Databricks if the JDBC driver can be included (ideally yes, but we can move forward even if not). This code also handles bulk insert via DBFS (Databricks File System).
Testing data quality R packages
ETA: November 2021
Thus far, we have Achilles working, but DataQualityDashboard is pending.
Testing PLE HADES R packages
ETA: January 2022
CohortDiagnostics and CohortMethod will be prioritized here.